Model calibration is the process of ensuring that the predicted probabilities produced by a model align with the true underlying probabilities of the data. A well-calibrated model will output probabilities that accurately reflect the likelihood of a certain outcome. For example, if a model predicts a probability of 0.8 for a positive outcome, it should be the case that 80% of the time the outcome is actually positive.

A binary classification model can have high precision and high recall, but if the predicted probabilities produced by the model do not align with the true underlying probabilities of the data, it is not considered well calibrated. An example of this can be a model used for predicting credit default, which is utilized by financial institutions to determine the likelihood of a borrower defaulting on their loan. These models are usually trained on large datasets of historical credit data and various features such as income, credit history, employment status, and more.

However, if the model lacks proper calibration, it may not accurately predict the probability of default, despite being able to accurately separate positive from negative examples. For instance, the model may predict that a borrower has a 90% chance of defaulting, when in reality the probability is only 50%. This discrepancy can lead to poor decision-making, as the model's predictions will not be in line with the true probabilities, resulting in financial losses for the institution and adverse consequences for borrowers.

I recently wrote about a price optimization model that I had developed in the past. In that particular model, it was also critical to ensure we had calibrated scores, as any inaccuracies in the predicted probability of a sale would have led to flawed calculations of the expected revenue (sale probability * potential revenue = expected revenue), ultimately resulting in suboptimal pricing decisions.

To ensure the accuracy of the probabilities and sound decision making, it is often important to calibrate a model. Techniques such as Platt scaling or Isotonic regression can be used to adjust the model's predictions to align with the true probabilities. Platt scaling is a technique that involves fitting a logistic regression model to the predicted scores of a binary classification model, while Isotonic Regression involves fitting a free-form line to the predicted scores. Both techniques aim to discover a transformation of the predicted scores that aligns them more closely with the true probabilities.

In conclusion, model score calibration is a crucial step in ensuring that the predictions made by a model align with the true underlying probabilities of the data. A well-calibrated model will generate probabilities that accurately depict the likelihood of a specific outcome. It is important to consider the usage of the model scores and implement calibration during the development and deployment stages of machine learning models when necessary.

The Business Solutions Series is a compilation of solutions to various business challenges that I have encountered throughout my professional journey.

Context

We were engaged by a leading clothing company that operates hundreds of stores worldwide and is renowned for its affordable custom clothing designs. The company sought to accurately forecast the number of clothing items, in various colors, they would sell in advance of each season.

Problem

The company updates its clothing lines and colors every three months, making it crucial for them to accurately forecast sales in order to optimize revenue and avoid excess inventory. The company needed to forecast sales several months in advance to order the necessary fabric and produce clothing with their manufacturing partners before each season began.

Objective

To provide the company's planners with highly accurate quarterly sales forecasts for all items, in all colors, to enable optimal production decisions.

Solution

To address the challenge of accurately forecasting sales for the clothing company, we utilized an AutoML managed service to develop a complex regression model. This model outperformed the more traditional forecasting methods that the company had previously employed. While the company only required quarterly forecasts, we found that training a more granular model and aggregating simulated numbers for the quarter resulted in more accurate predictions.

We utilized the company's historical sales data to train the model to make predictions for each day, store, item, and color. Our model included three types of features: date features (day of the week, day of the month, month of the year, holiday, etc.), store features (country, city, store size, etc.), and item features (type, subtype, color, standardized description, etc.). Using the new items in new colors, we then used the model to simulate sales on all stores for each day in a future season and aggregated the data to the appropriate granularity for the planners.

To ensure the model's real-life accuracy given the granularity of our training data, we were careful to prevent data leakage between the train, validation, and test datasets by splitting them by quarter. This ensured that the model was validated and tested on full quarters with new products/colors that it had never seen during training.

One of the biggest challenges was convincing the planners to trust the model's predictions, even when we had data to prove its accuracy and superiority. To address this, we implemented what-if simulations, using items from previous seasons to forecast sales for future seasons on these past items. This helped the planners identify which new items the model predicted would sell similarly to past items/colors, increasing their confidence in the model's predictions. Additionally, the model allows the planners to run simulations of new items on past seasons, providing further insights on how they would have performed.

Impact

The implementation of these new models had a significant impact on the company's operations. The planners began utilizing these models for their decision-making, resulting in millions of dollars in revenue and significantly fewer leftovers.

As an AI enthusiast, I'm always excited to explore the latest developments in machine learning. Today, I want to share my thoughts on what I believe is the next frontier for AutoML - custom embeddings.

In a previous blog post, I talked about my preferred AutoML tool - vertex AI AutoML. While it's a great tool to build ML models, it won't let you learn embeddings from your data. I believe that the future of machine learning lies in the development of tools that allow users to learn custom embeddings from their own data. These embeddings can represent different business entities and have the potential to transform the way we solve business problems.

So, what exactly are embeddings? In a nutshell, they create a mathematical representation of different business entities such as users, products, and stores, allowing computers to understand information about these entities and how they are related to each other. This is incredibly useful for businesses, as embeddings can be used to build ML models with very sparse training data and also improve the speed of model inference in high-volume real-time systems with very strict latency requirements.

In my opinion, the Two Tower model is a viable solution for addressing this issue. It is a powerful neural network architecture that can create high-quality embeddings of various business entities. The model embeds pairs of diverse entities, such as users and products, which are each represented by structured and unstructured data, into the same embedding space. This allows the embeddings to encode how different entities relate to each other. However, there is currently no robust AutoML-managed tool on the market to train a Two Tower Model, making it difficult for businesses to adopt this technique without substantial expertise and manual effort.

Fortunately, the field of AutoML is constantly evolving, and promising developments like Graph Neural Networks (GNNs) are also emerging. Kumo, for example, is developing tools that model existing structured business data as graphs and create powerful ML models based on those graphs. Graph networks are ideal for learning potent embeddings for business entities in the data because graphs can easily encode complex relationships and dependencies between these entities. I am eager to see what the team comes up with in the form of new products.

In conclusion, the potential of learning custom embeddings using AutoML is vast and has the ability to transform the way businesses solve problems and innovate. Although there are currently no robust AutoML-managed tools available for creating custom embeddings, promising developments like the Two Tower model and Graph Neural Networks are emerging. As AutoML continues to evolve, we can expect to see new and more effective tools that will enable businesses to easily learn and utilize custom embeddings to their fullest potential.

The Business Solutions Series is a compilation of solutions to various business challenges that I have encountered throughout my professional journey.

Context 

We were engaged by a client that specializes in digitizing documents. The client sought a solution to enable their mobile users to capture physical documents using their phone cameras and fill the documents electronically through their mobile application.

Objective

The objective was to develop a model that could accurately detect all inputs on a document, including the precise location, size, and type of the inputs (e.g. name boxes, date boxes, signature boxes).

Solution

To achieve this objective, we utilized several of Google Cloud Platform's AutoML managed services and created a scalable python API (also utilizing a GCP managed service) to coordinate the calls to all of the managed services. The mobile application was able to call this custom API every time a user captured a document using their phone camera.

The custom python API received the document image as input and:

• Utilized the GCP Vision API, which is trained by Google, to extract all text and text metadata from the document image (text metadata includes the precise coordinates of each word in the document image).

• In parallel, utilized the GCP AutoML Image managed service where we had trained a custom object detection model and deployed it as a service. This model detected all input boxes on the document image and returned their bounding boxes (coordinates).

• Inserted a custom keyword for each detected input box into the document text (using the coordinates from both the Vision API and custom object detection model output to insert the keyword in the correct location).

• Utilized a final custom AutoML Text model, trained to perform entity extraction, to classify the input boxes into their respective types based on the surrounding text.

For example:

• Input document image:

  

• Vision API output:

  • Text: “Sample Document Name Date Signature End Of Sample Document”

  • Metadata word coordinates: Sample <0, 0>, Document <9, 0> , Name <0, 4> …

• Object detection model output:

  • Coordinates: BOX_1 <6, 4>, BOX_2 <21 ,4>, BOX_3 <0 ,7>

    

• Modified text using coordinates from both outputs: 

  • “Sample Document Name INPUT_BOX Date INPUT_BOX Signature INPUT_BOX End Of Sample Document”

• Entity Extraction model output when providing the modified text:

  • INPUT_BOX [0]=name,  INPUT_BOX [1]=date,  INPUT_BOX [2]=signature  

In order to train the object detection and entity extractions model, we had to first obtain high-quality labeled data. To do this, we collaborated with a data annotation company that utilized human annotators to label thousands of document images. The annotators drew bounding boxes around the input boxes and selected input types for these, providing the training data our model needed to learn. Once the labeled data was obtained, it can be loaded into the managed AutoML Image service with ease, allowing for the training of an object detection model. The default settings and robust capabilities of the AutoML managed service provided us with an effective model for this task. Initially, before we used professional help, we encountered some issues with the model, but upon realizing that the issue was with the training data which had some inconsistent annotations, we enlisted the services of a professional data annotation vendor. This led to an improvement in the quality of the training data and subsequently, an improvement in the performance of the model from the AutoML service.

Similarly, we utilized the input type annotations to train the entity extraction model using the AutoML Text service, which also resulted in the managed service automatically learning a model that effectively addressed this task.

While there are multiple companies that offer labeling services for machine learning projects, I highly recommend Sama due to their commitment to hiring individuals living below the poverty line in developing countries, primarily in Africa, and helping them improve their livelihoods. I have first-hand experience with Sama as my spouse worked there for several years. I had the opportunity to visit their centers, meet their agents, and read the academic study about their impact

Impact

As a result of this solution, the mobile application was successfully deployed and is now being utilized by hundreds of thousands of users globally to efficiently digitize and fill documents. The API has proven to be a valuable tool in streamlining the document digitization process.

A/B testing is a widely employed statistical methodology used to compare the performance of two variations of a product or system, designated as version A and version B. This technique is used to determine which version performs better by showing one version to a subset of users while exposing the other version to a different subset of users. For example, an organization may wish to evaluate whether a redesigned website leads to an increase in sales compared to the previous design and conduct an A/B test to measure the conversion rate of each group.

One important statistical measure utilized in A/B testing is the "p-value". The p-value is used to determine the likelihood that any observed differences between the two groups being compared are the result of random chance rather than a real difference in performance. A low p-value (generally below 0.05) implies that the observed difference is statistically significant and is unlikely to be the result of chance. Conversely, a high p-value (above 0.05) suggests that the difference may not be statistically significant and could have occurred by chance. It's worth noting that the p-value threshold of 0.05 is widely used as a convention in many fields, but it's important to keep in mind that other p-value thresholds such as 0.01 or 0.1 may be utilized depending on the context and desired level of confidence.

For instance, let's consider a scenario where an organization wants to evaluate whether a new marketing campaign will be more effective in increasing sales compared to the current campaign. The organization decides to conduct an A/B test by exposing half of the customers to the new campaign while the other half is exposed to the current campaign. The organization measures six different effects: the number of clicks on the campaign's landing page, the number of purchases made, the average order value, customer satisfaction score, repeat purchase rate, and overall revenue. Utilizing a p-value of 0.05 to determine statistical significance, the organization finds that the new campaign performed significantly better than the current campaign in terms of repeat purchase rate, but there was no significant difference in the other five effects.

It's crucial to recognize that when conducting A/B tests and measuring multiple variables, the likelihood of observing any statistically significant difference due to random noise instead of a real difference increases. If we take the example above with a threshold of 0.05 for the p-value and the measurement of six variables, there is a 26.5% (1 - 0.95^6) chance of measuring a "significant" difference due to random noise in any experiment. Given that one out of every four experiments can yield "significant" differences due to random noise alone in this example, can we truly be certain that the new campaign is superior? I am not so sure. This probability increases to 51.2% when measuring 14 variables. To address this pitfall, it is essential to carefully choose and measure just a limited number of key variables that are directly related to the hypothesis of the experiment, as well as consider using a different p-value threshold depending on the context and desired level of confidence.

In conclusion, A/B testing is a powerful tool for evaluating the performance of two variations of a product or system. However, it is essential to understand the limitations of the p-value, the increasing probability of observing false positive results when measuring multiple variables, and the importance of selecting only relevant variables and choosing the appropriate p-value threshold. By doing so, organizations can ensure that their experiment results are accurate and their actions based on these are sound.

The Business Solutions Series is a compilation of solutions to various business challenges that I have encountered throughout my professional journey.

Context 

We were working on a system that processes millions and millions of sales opportunities daily and needs to make optimal pricing decisions for each one. For each opportunity, the system needs to choose a price that will maximize revenue while considering the probability of the sale being completed at that price. As the price increases, the potential revenue from the transaction increases, but the likelihood of making the sale decreases.

Objective

The objective was to maximize revenue through smart pricing. 

Solution

To achieve this, we first discretized the range of product prices by dividing them into X equally spaced price options. We then used various transaction attributes (such as details about the product, the buyer, and the market conditions) to build a binary classification model that would predict the likelihood of a sale being completed at a given price. The price, along with the transaction attributes, were used as features in the model.

One of the crucial elements of this project was obtaining reliable and unbiased data to accurately learn the true probabilities. Given that the system handles a vast number of transactions daily, we utilized a small random subset of these transactions, and we randomly selected one of the X different prices for each transaction. We then used this set of randomly chosen transactions and prices as the training data for our model.

After obtaining a clean dataset for training the model, we leveraged an AutoML managed service to effortlessly learn a sophisticated yet precise neural network ensemble. This model generated accurate probabilities of closing a sale for any of the X prices in combination with the transaction attributes. For every new transaction, we generated X different sale probabilities, and then computed the expected revenue and selected the price that would maximize it (i.e. sale probability * potential revenue = expected revenue).

Impact

This straightforward solution allowed us to quickly implement the price optimizer in production and rapidly increase company revenue by millions of dollars.

I recently ran a small online advertisement experiment with the goal of  increasing the number of subscribers to this Substack publication, and also to write this post about the experience. I focused my experiments on two advertisement channels: LinkedIn and Google search.

LinkedIn was an obvious choice as this is the only social network where I (and also this publication) have presence and followers. It is also a social network focused on professional networking and professional topics which fully aligns with the essence of this publication.

I picked Google as well because it’s not targeting specific people based on their profile details and historical usage of a social network, but rather targeting anyone using a search engine who is trying to find very specific information on the Internet at that specific point in time. Google’s strategy is very different to targeting specific people in social media so I thought it would be interesting to try both channels and compare results.

Both of these advertisement channels have some key differences and similarities which I will try to explain below:

Billing

LinkedIn charges you for the total number of ad impressions served.

Google will charge you only for the clicks made on your ads regardless of how many impressions they end up serving.

Campaign Optimization Goal

LinkedIn will charge more per impression depending on your campaign goal (e.g. brand awareness, website traffic, etc.) They add different features to the ad depending on your goal (for example some campaign goals will add a “follow” button to the ad and some goals would not). However, in my experiments, picking any goal other than the most basic one had no impact on clicks or conversions but made the cost per impression (and the campaign in general) substantially more expensive without any benefits.

Google lets you optimize your campaign on clicks or conversions without any extra cost. To make my campaign optimize for conversions (my real advertisement goal) I needed to set up a Google Analytics account in Substack, link my Google Analytics account to my Google Ads account, and create a “goal” in Google Analytics to track every time it detected a “Signup” event in the website. 

Audience Targeting

LinkedIn provides an automatic way to generate “Lookalike” audiences on their network using their proprietary data and models. I leveraged this feature using all the users who have interacted with my publication’s LinkedIn page as an input. Their service then created a big audience of users in their network who are “similar” to my input users. They don’t disclose much about how these models work but I’m sure they use some sort of user similarity scores based on their profile information and their behavior on the platform.

LinkedIn also has an “Insights Tag” that you can add to your website/publication directly and build the “lookalike” audiences based on the visitors to your website. However, I wasn’t able to try this feature as the only way to set this up in Substack is with Google Tag Manager and, unfortunately, Substack’s implementation of Google Tag Manager code is broken (I have reported this to Substack but they said that this is currently not a priority.)

To set the targeting in Google I provided some relevant search keywords related to my ad (I started with “learn data science”) but it also has an automatic feature that lets Google experiment and suggest other keywords that will work well with your ad. It does this as it’s running the campaign so I let it run for a couple of days to gather all their keyword recommendations and use those too for my campaign. They ended up recommending more than 300 more keywords that I added to the campaign (e.g. “ml and ai”, “be a data scientist”, “path to become data scientist”, etc.)

Partner Ad Networks

LinkedIn lets you expand the reach of your campaigns by targeting specific LinkedIn users outside of LinkedIn (by using display ads on other websites). They achieve this by partnering with ad networks and publishers (other websites) in the industry. The cost per impression is reduced substantially when your ads run there so I also experimented with this feature.

Google also lets you expand your campaigns with two types of partners. Search partners, who also operate websites that are search based (users inputting a query to find information), and ad network partners which lets them run your (display) ads in random websites around the world. The cost per click also decreases when your ads run with partners so I also experimented with these. The search partners made a lot of sense to me as the targeting mechanism is similar to Google search, but the ad network partners didn’t as we aren’t really targeting a specific user in those random websites like we are with LinkedIn. That said, Google still charges per click and not per impression in those random websites so I was still interested in trying it out.

Ads

In LinkedIn I set the campaign to use all the postings I’ve made on my publication's LinkedIn page as ads.

In Google I gave them a few headers and descriptions about my websites and they ran experiments to find the best combinations of these for different queries.

Experiment results

Experiments

• LA: LinkedIn campaign leveraging ad network partners.

• L: LinkedIn campaign only (no partners)

• GSA: Google campaign leveraging search and ad network partners.

• GS: Google Campaign leveraging search partners.

• G: Google campaign only (no partners)

Metrics

• CPM: Cost per mille in dollars (cost per thousand impressions - industry standard)

• CTR: Click through rate or percentage of impressions that lead to a click.

• CPC: Cost per click in dollars

• SUR: Signup rate or percentage of users that sign up to the publication after clicking on the ad.

• CPS: Cost per signup in dollars.

Except for Google search, it is common for all publishers (websites), social networks, etc, to charge based on impressions. As a data scientist with experience in ad tech I’m not thrilled about having to pay for impressions. Impressions will be rarely correlated to my campaign goals (publication signups in this case) and there is no guarantee all these impressions are really being optimized for my benefit or, even worse, that some nefarious entities or users in the complex advertisement network are committing ad fraud.

Lets compare the experiment results using the CPM lens:

GSA experiment gave us the lowest CPM compared to all others. It is also interesting to see that although the Google experiments seem to have lower CPM in general, there is not a clear winner platform-wise as LA has a lower CPM than G.

As discussed earlier, Google charges based on clicks which in my opinion is a more valuable metric for advertisers than impressions. Clicks can sometimes be somewhat correlated to the campaign goals but unfortunately clicks are still far from perfect. Clicks can still not be correlated at all with the goal in some campaigns and fraudsters can also abuse these.

Lets compare the experiment results using the CPC lens:

GSA is still the clear winner also when using this metric. Note that based on cost per clicks the Google platform is also a clear winner over the LinkedIn platform both in terms of cost (which is not surprising to me given LinkedIn charges per impression and Google per click) but also in percentage of users clicking on impressions. Another interesting thing to note is that G has the higher CPC for the Google experiments but G also has the higher click rate, which shows that impressions on the Google network are a lot more relevant to the users than impressions that show up on the partners. However, those impressions come at a higher click price whenever the user clicks on them.

Let’s now focus on signups, which is what I really care about for these experiments. I don’t care about impressions or clicks if these are not leading to signups at the end of the funnel.

Lets compare the experiment results using the CPS lens:

Note how GSA, our previous CPM and CPC winner, is now at the bottom of the Google experiments with GS now being the winner on cost per signup as well as signup rates. GS is closely followed by G. Also note how G and GS had a huge difference in CPM but in terms of signups they both perform very well (which is what I as the advertiser really care about).

GS is the clear winner here in terms of how much I need to pay per signup on this campaign. This makes sense to me as this experiment leverages search keyword  targeting while also displaying some of the ads on cheaper (per click) search partners. The google advertiser network in GSA is not adding value and losing us money as I originally speculated. 

Unfortunately, in my experiments, LinkedIn was way more expensive and much less effective than Google. I was also disappointed that my following on that social network didn’t grow much with this campaign. This is the social network where I have a presence and I assumed that some extra advertisement could easily expand that following. However, this was not the case. 

On the other hand Google was a very effective channel. They had the lowest cost per click by far, the highest click through rate by far, and more importantly, substantially higher signup rate and substantially lower signup cost, which is all I really cared about.

I hope this article is helpful for when you are evaluating your own ad campaigns.

The Business Solutions Series is a compilation of solutions to various business challenges that I have encountered throughout my professional journey.

Context 

Improve a service to detect bots that engage in online ad fraud.

Problem

This problem is - for the most part - an unsupervised ML problem (e.g. we don’t have ground truth for bots). I like this project for this reason, and also because we ended up building different solutions that complemented each other. This allowed us to improve the solution over time.

Objective

Identify online bots. This is valuable for internet companies because it can substantially reduce the money that brands lose to ad fraud and also protect their services from malicious players without affecting their real users.

Solution

The first model our team developed was very simple and elegant. Some important things to understand: bots tend to be programmed to visit the same set of fraudulent websites (to commit ad fraud). And, these fraudulent websites are usually not visited by real human beings. This creates an interesting conditional probability distribution for those fraudulent websites.

First, we computed independent scores for all websites by calculating the probability that any random user will visit the site at any point of time. Popular websites visited by many users (e.g. facebook.com) would get a high score (this is the probability that any random user will visit it) and obscure websites would have a very low score (a random user is very unlikely to ever visit it). Then, for all pair of websites, we also computed the conditional probability of a random user visiting a website given they have already visited the other website in that pair.

Even though fraudulent websites will have very low probability of being visited by a random user, their conditional probability will be very high if those users have already visited other fraudulent websites before. These kinds of bots tend to visit mostly fraudulent websites so their browsing history will be heavily represented by website pairs like this one (pairs where each site has low scores independently but very high conditional probability when seen together).

For each user, we looked at all their browsing websites and also at all of their website pairs. If a user had mostly visited websites that had low probability scores but high conditional probability scores given most seen pairs, there was a very high chance that that user is actually a fraudulent bot. We encoded this logic into the data warehouse, and that helped us easily flag these users moving forward.

Initially, another easy way to catch bots was to look at browser type and version distributions between known legit websites and known fraudulent websites. Many bots are actually operating inside infected machines and use malware built-in browsers to do their fraudulent browsing under the hood, without the machine user even noticing this is even happening. By monitoring known fraudulent websites we could flag browser types and versions that seem suspicious compared to browsers types and versions seen in legit websites. However, fraudsters rapidly realized this was giving them up so they started faking (spoofing) their browser user agents to avoid being flagged so easily.

We then decided to start collecting a lot of information (hundreds of signals) using custom JavaScript code on those browsers. This was so we could detect browsers that were pretending to be some other browser that they were not. We initially started this project by hard-coding rules on those signals but very fast realized it was a management nightmare. We needed to keep up with new rules for new browser versions all the time. So, instead, we replaced that process with a machine learning model, where we pre-filtered browsing examples that we were very certain to be clean and trained a model to classify the browser type and version based on the hundreds of signals (features) that we were able to collect form each browser using our JavaScript probe. We then used an AutoML managed service to train this model. Given how comprehensive the amount and diversity of signals we were able to collect from browsers running our code, the ML model managed to do an amazing job at predicting the browser type and version - with 100% accuracy - on the “clean browsing” hold out set. We then used the same model to predict the browser type and version in the wild and started flagging those browsers where the user agent didn’t match the predicted browser type or version.

Lastly, we also complemented the solution with one more model that helped us separate cloud computers that were being used legitimately to browse the Internet (e.g. a VPN service) from those being used fraudulently (e.g. bots using cloud VMs). For this use case, we got our hands on some labeled data to train our model. We found a big list of cloud IPs that we knew for sure were being used by VPN-like services and some other big list of cloud IPs that we were very confident we shouldn’t see any real human traffic on them (but we were still seeing it).

Both kinds of cloud IPs would show a high volume of browsing coming from those IPs but we were able to find a few features that helped us train an almost perfect model. We realized that cloud IPs for which no human traffic was expected would have many short lived non-overlapping cookie sessions (bots were likely clearing all cookies after every website visit), while VPN-like IPs would show many concurrent cookie sessions coming from the same IP with a unique distribution of how long different cookie sessions lasted (as real people were actually behind those sessions). By computing counts of total cookies, maximum number of concurrent cookies during some time window, and some metrics about the distribution of the cookie’s lifetime, we got an AutoML managed service to learn to differentiate these two kinds of Cloud IPs beyond the smaller sample we had labels for.

Impact

By combining many kinds of models to tackle this problem we were able to build one of the most effective bot detection solutions in the industry.

The Business Solutions Series is a compilation of solutions to various business challenges that I have encountered throughout my professional journey.

Context & Problem

Customer service representatives answer calls from thousands of clients a day. The costumer success team wanted to understand how often different issues came up on those calls and what exactly was being said about those issues by the clients, and by the costumer service representatives who answered those calls.

The team needed an efficient and flexible tool to analyze thousands and thousands of recorded phone calls for different issues. 

Objective

The objective of the project was to process recorded telephone calls to rapidly help identify different topics and issues throughout many conversations. This was needed to understand the prevalence of different issues, the nature of those issues, and how those issues were being handled.

Solution

To solve this problem we were able to leverage TensorFlow Hub, which is a free and public repository of pre-trained machine learning models. We were particularly interested in their natural language (text) models. They have several of them including some very interesting multilingual ones that can embed phrases from 16 or more different languages into the same embedding space (e.g. the same phrase in different languages ends up having very similar embeddings). These models are extremely easy to use, as you can just load them into Python and use them as a function that takes a phrase (string) as an input and outputs an embedding for that phrase. You can also make use of these models inside TensorFlow Serving if needed (to easily speed up and scale inference).

For our specific problem, we set up a managed Apache Beam job that would run a few times a day to process all new calls. This process called a managed cloud transcription service that returned separate strings every time the service detected a new speaker or a long pause on the audio files. Our job was to then take each of those strings and run them through the pre-trained language model of choice, which outputted an embedding for that phrase. Then, we saved the transcript string, the call identifier, the start time of the phrase, the duration of the phrase, and the embedding of the phrase into a data warehouse.

Lastly, we let users define topics and issues of interest (by writing a description of the topic in plain english) and we used the same pre-trained model to convert those descriptions into embeddings. Once we have the topic embeddings and the transcript embeddings, we can easily use a zero shot model (basically embedding cosine similarity) coded directly inside the data warehouse to find all the phrases in the transcripts that were semantically similar to a topic of interest, and show these to the end user for their analysis.

Impact

The company was able to easily understand the true scale of different issues, and find out what clients were saying about specific issues and how representatives were responding to them. This lead to better issue prioritization and robust solutions to these common issues, which then reduced substantially the number of calls the team was receiving and increased client satisfaction over time.

The Business Solutions Series is a compilation of solutions to various business challenges that I have encountered throughout my professional journey.

Context 

The goal of this project was to protect acquiring banks - banks that provide credit card terminals to businesses - from nefarious merchants. These are merchants who use their online terminals to transact on products and services that are not allowed by the terms of the contract with their bank (e.g. counterfeit products, illegal substances, etc.)

Problem

The way these fraudulent merchants were operating is by sending bank applications misrepresenting the nature of their business in order to be approved. Once approved, they used the online terminals on their websites to sell prohibited products. 

In order to crack down on this practice a team of analysts was going through all the merchant applications to verify that their listed website was not selling any prohibited product/services (and flagging those that were). In the early days it was very easy to detect thousands and thousands of these bogus applications by only going to their listed websites. But, as soon as they realized it was easy to monitor them, they started creating professional looking front websites in which they pretended to sell allowed products. However, their real intention once approved was to still use the online terminal on an undisclosed website that was breaking the terms.

Objective

To prevent nefarious merchants from using credit card terminals to sell prohibited products. 

Solution

We had a rich labeled dataset of websites (several tens of thousands) that sold prohibited products from the early days. The data set consisted of a text document (text parsed from those websites with a crawler) and the different labels that our analysts provided when flagging them for prohibited products.

We were able to feed this data directly to an AutoML managed tool that would automatically pre process all the text into 1-grams and 2-grams (lists of all words and list of all pairs of words) and will learn embeddings for all these n-grams. It was also able to find the best neural network architecture and hyper parameters to train a very accurate multi class classification model. Once we had this model, we piggy backed on an open source project called Common Crawl which periodically crawls the whole internet and saves the crawled data into a public cloud storage for anyone to read. We then spin up a managed Apache Beam job to process the common crawl data and run each crawled website text through our model. Lastly, we dropped all the URLs with predictions into a data warehouse for the analysts to query later.

We then armed all our analysts with credit cards that were set up to always be declined by the issuing bank. We did that to track the merchants and acquiring banks on the other side of the transactions. By using the scores in the warehouse (prioritizing those with higher scores for prohibited products) our analysts were able to quickly find thousands of online stores selling prohibited products and taking credit cards without their bank knowledge. We then attempted to make purchases in those sites in order to find which specific merchants were using their online terminals on websites they didn’t list on their applications. These merchants were later shut down for breaking the terms.

Impact

The main takeaway from this story is that because we already had a very rich training dataset and the Common Crawl data at our disposal, we were able to easily leverage a set of managed services to produce very good business results. We only needed the initial model to have a low false positive rate on the top-scored websites, and the AutoML tool managed to do that very well. The scores from this model were then used to rank websites for analyst prioritization which allowed the analysts to focus almost exclusively on nefarious merchants and rapidly process and shut down thousands and thousands of them.

For this post, I am testing three new out-of-the box AI tools that will help me easily and quickly put together a complete data science post for my publication.

The first tool I'll introduce is DALL·E 2 from OpenAI. This new AI system can create photorealistic images and artwork by simply reading a description given in natural language.

So far, I've used this tool to create all illustrations for my Substack posts, including this one.

With this tool, you can create art by simply typing a description of the image you want to generate. For example, for the illustration in this article, I typed out the following:

“Three robots, one drawing a painting, one writing on a notepad, and one playing with dice”

The tool produced these four image options from my prompt:

While I liked the third option best, it had a few issues. The drawing included four robots instead of three. Additionally, none of them appeared to be playing with dice, and it wasn't evident which robot was writing and which one was drawing.

The tool also includes an editing function that allows you to delete certain parts of any illustrations and change the prompt. This generate 4 more options based on your edits.

I deleted the following from my favorite illustration:

And modified the description to say:

“A robot writing in the bottom left, a robot painting on the bottom right, and a robot rolling dice in the back”

The following 4 options were generated by the tool:

The tool’s third option was my favorite, even though it added back a fourth robot. At least the two robots in the back are now clearly playing with dice, and with the two robots in the front it is now clear which one is writing and which one is painting. I ended up using this as the illustration for this post.

The second tool on my list is GitHub Copilot. It's a fantastic time-saver that allows you to code in multiple programming languages, including Python, by simply using natural language prompts.

In a post on my publication about the probability of earthquakes in Mexico, I used some basic python code to run simulations in order to answer the question posed by the article. For this new post, I want to use GitHub copilot to write the python code. This tool will automatically translate my English language instructions into working python code.

I started by creating an empty file called “simulation.py” and typed the following comments:

# sample 64 days from 1 to 365
# and assess if any day was sampled three or more times
# if so, return true
# If not, return false

As soon as I hit Enter, the tool suggested to add the following Python code to the file:

def simulation():

   days = np.random.choice(np.arange(1, 366), size=64, replace=True)

   return np.any(np.bincount(days) >= 3)

I was impressed by this tool and accepted the recommended code, but I didn't know that it was just getting started. When I pressed enter again after the tool generated its code, it automatically suggested adding these new comments:

# Path: main.py
# run simulation 10000 times
# and calculate the probability
# that at least one day was sampled three or more times

The comments impressed me as they were exactly what I wanted my program to do next--I didn't even have to write them. The tool even accurately guessed that I wanted to run the simulation ten thousand times. It was quite spooky.

I hit enter, and the tool immediately created the rest of the code it required from its own comments.

def main():

   simulations = [simulation() for _ in range(10000)]

   print(np.mean(simulations))

if __name__ == '__main__':

   main()

Running that code produced the same results as my original code, except it was quicker to write and neater. All I did was type a few short comments and then press enter repeatedly to accept all the recommendations--the whole process only took me less than 2 minutes.

In addition to the aforementioned tools, I'm also using Jasper to help create this post. For those who don't know, Jasper is a high quality AI copywriting tool that eliminates creative blocks so you can write original content faster.

I hastily typed this article into their tool and the results were a much better quality text. In fact, all of the text you've read in this post has been edited or rewritten by the tool.

For example, here is the original text for this last paragraph that the tool completely rewrote:

"The last tool I’m using to produce this post is called Jasper. Jasper is a high quality AI copywriting tool that helps break through creative blocks to create amazing, original content faster. The text for this article I quickly and carelessly typed into their tool and got the tool to edit and rewrite the text in much better shape. All the text you’ve read so far for this post was edited and sometimes fully rewritten by the tool"

In less than two hours, I learned how to use three different tools and produce the content for this post. It's fascinating that we've reached a point in AI technology where very good but not perfect vision and language models can be leveraged by tools that help people complete their tasks much faster and efficiently.

The Business Solutions Series is a compilation of solutions to various business challenges that I have encountered throughout my professional journey.

Context 

We were working with a company that sells and then services elevators for buildings all across the world.

Problem

Elevator technicians are expensive to hire and their time is very valuable. When an elevator breaks down and the technician is called, oftentimes they won’t know what is really wrong with the elevator until they are onsite. Also, more often than not, they won’t have with them all the parts they need to fix it (as it is impossible for them to preemptively bring all possible spare parts with them on all of their visits). Hence, they will have to arrange to come back at a later time with the right parts to fix it (and do it soon as elevators need to be fixed soon per the terms of service).

Objective

To reduce elevator servicing costs and decrease downtime of broken elevators.

Solution

The company we were working with had implemented a system on many of their elevators that would send all elevator system errors to their cloud service. Initially, they thought these reports were noisy and useless as elevators produce dozens of error codes all the time, and most of the time, nothing is really wrong with the elevator. 

However, they also had technician reports of what exactly was wrong with an elevator every time they had to go fix one. 

By using the technician reports together with the error system logs, we were able to identify sequences of errors (sequences of length 2 - 5) that were significantly correlated with instances where specific problems happened. We then picked the top 200 of these sequences and created a training set with the technician reported problems together with the counts of times each of those pre-selected 200 sequences were in the logs for a few hours before the elevator was reported as broken. We then trained a multi class classification model (using an AutoML managed service). Lastly, we made changes to the servicing process for the technicians to bring spare parts for the top 10 most likely failures - based on the model predictions - every time they went to check on a broken elevator.

Impact

We were able to substantially reduce the number of instances where the technicians -on their first visit - didn’t have the correct parts to fix the elevator with them. Hence, reduce servicing costs overall. Note that for this business solution to produce substantial savings, the multi class classification model didn’t really need to be very accurate on its top prediction, it only needed to be very good at making sure that any of the top 10 predictions was the correct one most of the times.

The Business Solutions Series is a compilation of solutions to various business challenges that I have encountered throughout my professional journey.

Context 

We were working with computer factories all over the world, and these factories were being handed rolls of certificates that they needed to include with the computers. These rolls were being produced using technology similar to the one used to make money. Each roll had a roll ID and each certificate inside a roll had a certificate ID. These were being used to track their life and usage across the supply chain. 

The purpose of these certificates was for the end user to reactivate the software in case they needed to perform a factory reset on their device. Computers came from the factory with its software already activated, so most of the time these certificate codes were not used. However, a common exception was random hardware glitches that would require a factory reset.

Problem

The certificate codes were very valuable in the black market as they allowed people who didn’t purchase the software legally to activate pirated versions of it. This gap rapidly developed into a cat and mouse dynamic with some of the factory workers stealing codes with increasingly sophisticated methods, and the software company trying to stop the practice. 

Initially the workers were writing the codes in pieces of paper as they were taking each certificate out of the roll and putting it into the hardware box. The software company would just quickly block activation certificates once they started seeing more than a few usages of a single certificate. But then the workers started copying more certificate numbers so none of them would need to be reused more than once or twice in the black market. 

Later, the company was also able to identify rolls - with the roll IDs - that seemed to be fully compromised and flag the whole rolls. This way the certificates of those rolls couldn’t be abused anymore. This led to even more sophisticated code copying at the factories, where the workers stealing them would not steal all certificates from the same roll and would instead alternate between copying them or not as they were processing them.

Objective & Assumptions

The objective of the project was to stop the piracy practice of stealing certificates. 

We knew that legit factory resets do happen once in a while, but they were mostly due to “random” glitches in the hardware. We also assumed that humans stealing the codes were probably not using random number generators when choosing which certificates to steal (even if their code copy patterns had evolved to be more complex).

Solution

My manager introduced me to a statistical test called “runs test”. It is a simple test where we marked the certificates with a binary flag: if they have been used for reactivation or not. Then, we ordered the flag values into sequences of positive and negative runs - in the same order that the certificates came on each roll, where a run ends every time the consecutive certificate has a different flag value.

For example: A roll with 26 certificates and an activation sequence 00001110011110001001110000 has 9 runs.

The test consists of estimating the expected number of runs based on how many positive and negative values were seen on the roll under the hypothesis that each element in the sequence is independently drawn from the same distribution. If the actual number of runs is a few standard deviations away from the expected value, we would then flag those rolls as suspicious.

The formulas are so simple that it was very easy to implement into the SQL data warehouse and set up - right in the warehouse - a monitoring process to flag any suspicious roll moving forward.

If you are interested in learning more about this test this wikipedia article can be a good starting point.

Impact 

The new flagging mechanism worked like a charm. The company was able to find - very quickly - thousands of rolls that they didn’t know had been compromised. They were able to also monitor new rolls and quickly react moving forward. This had a very positive impact on revenue as piracy practices were massively reduced after this went to production.

As someone who's passionate about data science, I've discovered that one of the best ways to reduce my team's dependence on others is to utilize cloud managed services for our machine learning infrastructure. In a previous post, I mentioned the significance of enhancing software engineering skills within the team. However, I believe that deploying clusters, monitoring virtual machines, or writing intricate parallelization logic shouldn't be a priority when we have the option to utilize managed services that can do all of this for us. It's important to always be on the lookout for ways to enhance our work and make it more efficient.

Personally, I find Google Cloud Platform (GCP) to be the best cloud provider for machine learning work. Although I have experience building solutions on various platforms such as AWS, Azure, and on-premise Hive/Spark, GCP remains my top choice. Below is a list of my favorite managed services within GCP. For those who find some of the concepts difficult to grasp, there are plenty of blog posts and documentation available for further exploration.

1. Vertex AI AutoML. This managed service specializes in scaling automatic model training, feature engineering, hyper parameter tuning, and neural architecture search. In my experience with various AutoML tools, such as H2O Driverless, Turi (prior to its acquisition by Apple), Azure AutoML, and Sage Maker, Vertex AI outperforms them in terms of ease of use, scalability, infrastructure reliability, accuracy, and model flexibility in terms of utilization.

2. Vertex AI pipelines. This managed service allows end-users to create complex orchestration pipelines to train and deploy new models on a schedule. The creation of pipeline components is achieved through simple Python functions, which can then be mixed, matched, and reused to generate complex execution graphs. The managed service handles the execution of these pipelines on managed compute infrastructure, where each component of any pipeline runs in its own separate Docker image instance.

3. Big Query. A managed data warehouse (SQL) that separates storage from compute (similar to Snowflake) to scale significantly. This service has in-built scalable ML models (such as k-means, matrix factorization, simple regression) and robust integration with Vertex AI and DataFlow.

4. DataFlow. A managed service for Apache Beam, which aids in defining complex data processing pipelines with simple Python code. The managed service then scales and parallelizes these pipelines, without end-users having to be concerned with the complexities of scaling and parallelizing their job, or the compute clusters behind their job.

5. TensorFlow Hub. This repository offers free trained models for the transformation of text or image data into powerful embeddings that can be used as features in individual models (transfer learning).

In conclusion, cloud managed services have revolutionized the way data scientists work with machine learning infrastructure. The use of managed services can simplify and accelerate the development of new Data Science products.

Throughout my time in the world of data science and machine learning, I've had the pleasure of speaking with many people who are fascinated by this field. However, I've noticed that a lot of individuals become overwhelmed and doubtful about pursuing it after a short time. They often feel discouraged when they encounter the complicated papers on advanced neural network architectures and discover that those engaged in this kind of work generally possess highly specialized doctoral degrees, extensive education, and significant experience. As a result, they see the idea of acquiring the necessary knowledge and experience as daunting and almost impossible.

However, it's important to note that a complete understanding of every paper is not necessary to enter the field, and neither is a PhD. Advanced machine learning work is fascinating, and it's certainly going to be an influential factor in the future of the field, but the majority of Data Science teams - whether in large or small companies - don't require that level of specialization. It's not necessary to fully comprehend every paper to be effective in your day-to-day work.

There are several key factors that are much more important in becoming a successful data scientist, including:

1. Maintaining a curious and inquisitive attitude towards the business problems you are trying to solve. You'll need to learn how to clearly frame business problems as data science problems to tackle them effectively later. Additionally, it's crucial to identify the actual problem that needs to be solved - often, what the business thinks they want to solve isn't actually the issue. By identifying the correct problem from the beginning of the project, you'll be able to build successful and transformative solutions to real-world problems.

2. Cultivating a statistical and analytical mindset in everything you do, from how you collect data to how you frame problems. It's essential to be mindful of any biases that may exist in either the data or the framing of the problem. Additionally, be careful to avoid data leakage and ensure that you're assessing solutions based on what is truly valuable for the specific business problem.

3. Developing your software engineering skills, even if you don't have a computer science background. This means learning to code effectively in Python and SQL, and writing clean and decoupled code that is easily reusable and maintainable. Successful data science teams are those that can deploy their business solutions to production without depending on other teams. If you're new to this area, don't worry - the other two skills are more important, and you'll have the opportunity to learn and practice as you work.

Above all, it's important to enjoy the journey of learning and growing in Data Science. Don't be too hard on yourself and remember to stay curious and open to new challenges. By cultivating the skills mentioned above and maintaining a positive attitude, you'll be well on your way to a fulfilling career in Data Science.

Mexico had a major earthquake, again, on September 19th. This is the third major earthquake on this specific date (Earthquake Shakes Mexico on Anniversary of 2 Major Temblors).

Many people in the country are in awe of this, asking themselves how could something like this happen.

In fact, the probability of three major earthquakes (over 7.0 in richter scale) happening on the same date is not as low as you would think. It is actually about 25% likely to happen. This is the same probability as getting two consecutive "heads" when tossing a coin twice.

I was able to estimate this probability with a simple Python script. I simulated (100K times) picking random dates for the 64 major earthquakes we have had in Mexico (according to Wikipedia).

But why do many people think such an event is less likely than it really is?
The birthday paradox has a similar effect. You only need 23 random people to have a 50% chance of having two of them have the exact same birthday, but most people will not think this is as likely.

A friend of mine pointed out that two of the three earthquakes had substantial casualties. About 30% of major earthquakes in Mexico have had substantial casualties according to wikipedia (defined as having 25 or more casualties). I updated my script and ran a second simulation to compute the probability of having at least 3 major earthquakes on the same date, where at least 2 of those had substantial casualties. I did this by picking random dates and then picking a wighted random flag with 30% of chances of that flag being true.

The probability of that happening - of at least 3 major earthquakes in which at least 2 of them had substantial casualties on the exact same day - was still higher than 7%. This is several thousand times more likely than my wife winning a raffle for a fully paid trip to Maui in the Four Seasons (which she won), or her winning a raffle for a free Safari trip to South Africa (which she also won!).

Sometimes people, including data scientist, need to challenge their own pre-conceptions and biases.

Hello, my name is Alan and I am an experienced data scientist and machine learning engineer with over 12 years of experience. Originally from Mexico City, I am passionate about using data to solve complex business problems and develop innovative data products.

My interest in computer science stemmed from my childhood dream of becoming a software developer. However, during my college internship, I worked with statisticians and was intrigued by the models they built and the process of constructing them. That experience led me to take as many machine learning, data science, and statistics courses as possible during my undergraduate and graduate computer science programs, as well as after graduation.

In the early years of my career, I made a conscious effort to move away from general software development and focus on data science. I was fortunate to land a job at Microsoft, where I was mentored by a seasoned statistician and had the opportunity to learn from experienced practitioners who emphasized the importance of fully understanding business problems before developing solutions.

Throughout my career, I have been fortunate to have diverse experiences, from working for small startups in various industries to consulting for some of the biggest companies in different fields. I have also had the privilege of working under exceptional leaders who empowered and supported me, teaching me the value of prioritizing working with great managers, leaders, and colleagues.

Outside of work, I enjoy traveling with my partner Carla, indulging in delicious cuisine, snowboarding, hiking, and embracing all that life has to offer. You can follow our adventures on Instagram or on our blog .

This publication is a compilation of business problems that I have tackled throughout my career, with a focus on simple solutions. Its purpose is to share my experiences, thoughts, and high-level recommendations with those interested in pursuing a career in this field.

While this material does not delve deeply into implementation details or the use of recommended tools, ample documentation is available online.

My objectives in creating this material are threefold, and I hope to achieve at least one of the following:

• Debunk the notion that a PhD-level expertise in neural networks is necessary to succeed as a data scientist and solve significant business problems.

• Share simple solutions that have successfully addressed major business problems during my career, with the aim of inspiring others.

• Publish this work in the hope of transitioning from a traditional corporate career to a more flexible lifestyle that combines my passion for long-term travel with my passion for data science and machine learning. I am currently available for advisory opportunities with startups or large companies in this field. If you believe that we can collaborate, please do not hesitate to connect with me on LinkedIn (@alankrumholz) or send me a message.

Acknowledgments:

I would like to express my sincere gratitude to my life partner, , for her valuable editing contributions and unwavering support and encouragement towards my pursuits.

I would also like to extend my appreciation to the exceptional mentors, leaders, and data scientists who have been instrumental in propelling my career and collaborating with me on the various projects that will be discussed in this publication.

This is Data Science is not Rocket Science, a small publication about simple and effective solutions to real business problems.

Publication Content

Introduction

• What this Data Science Publication is and is not

• My Journey to Becoming a Data Scientist

• Tips for Succeeding in Data Science without a PhD

Business Solutions Series

The Business Solutions Series is a compilation of solutions to various business challenges that I have encountered throughout my professional journey.

• Stopping Software Piracy: Using Simple Statistical Test

• Efficient Elevator Repairs Using Machine Learning

• AutoML Stops Fraudulent Merchants

• Analyzing Customer Conversations with NLP

• Combating Ad Fraud with Advanced Bot Detection Solutions

• Optimal Pricing for Revenue Maximization

• Computer Vision: Frictionless Mobile Document Digitization

• Accurately Forecasting Sales Months in Advance with ML

Miscellaneous

• Cloud Managed Services & Favorite Machine Learning Tools

• Three Major Earthquakes in Mexico: Exploring the Probability

• How AI tools helped me create this data science post

• Online Ad Experiment: Comparing Ad Channels

• Navigate A/B Testing Pitfalls: Avoid False Positives

• The Next Frontier for AutoML: Learning Custom Embeddings

• Importance of Model Calibration: Techniques & Benefits