Dynamic Prediction of Trucking Freight Prices to Digitize the Indian Trucking Economy (Part 2)

13 July 2018
Technology

Freight Rate Exchange was conceptualized to make the process of price discovery in the trucking marketplace hassle-free and transparent. In Part 2 of this series, we dive into the architecture design of this engine.

Introduction

In the first part of this series, we had introduced Freight Rate Exchange (FRE) and the objectives for the same. FRE enables real-time freight information for 1 Mn+ lanes based on ODVT (origin-destination-vehicle type). This was envisioned to bring price transparency to the otherwise unorganized freight market.

Every shipper/carrier operates in different locations and lanes with various vehicle types. Therefore, it was important that we showed relevant personalized ODVT feed for every user. Enabling this would create real value for users and improve user stickiness on our network.

When a user visits FRE, we capture his/her actions. These actions could be lanes viewed, time spent on cards, price quotes etc. This information is then used as features in multiple AI engines to improve the system accuracy, in a self-learning manner.

 

FRE architecture

Following were the basic requirements for architecture design.

  • Provide relevant feed to the user
  • Prioritize highest reliability prices in relevant feed
  • Continuous availability of route prices to users
  • Seamless delivery of price feed to the user
  • Handle traffic for querying 1 Mn+ lane prices
  • Support ten regional languages
  • Set up a feedback system to enhance price data liquidity
  • Create a self-learning model to improve price accuracy

To solve for these requirements, we utilized the micro-services architecture and delegated tasks to appropriate services.

  • Backend system
    • Captures price updates provided by the pricing engine and delivers it to the user interfaces efficiently
    • Transmits user price feedback to pricing engine and ODVT recommendation system
  • ODVT recommendation system: Identifies ODVTs which are most relevant for any user through a personalized route recommendation engine
  • Pricing engine: Updates prices for each ODVT real-time

 

Backend framework

Price feed

For the price feed for any user (shipper/carrier), we needed to ensure that the recommended lanes for prices were relevant and their pricing was accurate. This meant that we show top lanes with the most recent prices. At the same time, we needed to show the price seamlessly without any delays / latency.

To ensure the price feed is always available, we persisted a pre-processed collection of the entire lane set with its address details in ten regional languages in Mongo DB. For customized user feed, we used Redis to store a set of routes for each user (using user route recommendation engine) as a cache that gets invalidated every few minutes.

ODVT price search

The user route recommendation engine is also a learning system and the accuracy of the feed is linked to the extent of user activity on product. Therefore, we decided to allow users to search price for any route. We used AWS-hosted CloudSearch service to enable search functionality. It stores information regarding all the lanes, provides location-based indexing and sorting of indexes on huge amounts of data efficiently.

Price feedback

Given that the reliability of the pricing engine was gradually improving, it was important to enable crowd sourcing of prices as well. It served the following three purposes.

  • Improved data liquidity
  • Allowed user interaction events
  • Enabled feed customization

Therefore, users were given the option to enter the prices on lanes. We delivered these inputs to our route recommendation engine using hosted services like AWS SQS and SNS. The input prices were used to dynamically update pricing engine prices within permissible variance norms to visually deliver impact of user’s price input event. This enabled better user engagement.

 

ODVT recommendation system

For every user, recommended ODVTs are generated by user profile creation. User profiles are created using user activity events like search, order clicks, loads view, load posts, price view etc. This will be covered in more detail in a future blog on recommendation engine.

For day 1 users where zero user activity exists, initial feed logic was defined as given below.

  • If the user enables GPS location sharing, most liquid lanes originating from location
  • If the user does not enable GPS location sharing, most liquid lanes at overall system level in the past week

Post initial feed, update of relevant ODVTs happen on real time basis using data from event handler. This is done using a caching mechanism for the profiles which help in making it a high throughput engine.

Our lane universe is not static and expands with every new order / user search event. This means that we need to inform consumer services (downstream services) regarding the newly added lanes regularly. This is done using AWS SNS subscriber publisher model. FRE consumes this information and updates relevant ODVT information.

Moreover, whenever a user quotes price and a price advantage exists for the user on platform, notifications are sent in real time to the user. Given the dynamic nature of prices, we cache ODVTs for which prices have been quoted and data is made available using event driven architecture.

 

 

Pricing engine

Pricing engine receives price inputs on an ODVT from backend infrastructure framework. These inputs are ingested using SQS and processed by derivative worker which generates derived updates. To know more about the architecture of the derivative worker please refer this blog.

On receiving price inputs from users, a reliability score for the quote is calculated. This is done by using machine learning models. More details about the model will be shared in subsequent blog posts.

Once the reliability of a particular user’s quote is determined, the next step is to decide whether to consider the quote’s impact on price or not. Entire time series of price inputs with respective reliabilities and recency-based weightage function is used to determine the new price point. The code snippet below demonstrates the exponential delay function in finding reliability of quotes.

 

 

Using this price, we create and modify price cluster spread for each ODVT and correspondingly update the price as the centre of the applicable cluster.

 

 

Overall architecture diagram

 

 

Impact

At RIVIGO, we solve real world problems and therefore, the delivered impact of any work is the most important yardstick to measure technology. We were able to provide a seamless and hassle-free experience to users by delivering price information for routes swiftly and precisely. Traffic on FRE has been on a constant rise proving that price discovery is a real challenge for users and hence, a solution like this was much needed. The number of FRE quotes received has also been on a constant rise. We received valuable inputs that improved accuracy of our recommendation and pricing engines.

In the next article, we will cover the approach and architecture of the Ramanujan Price Prediction Engine, which uses machine learning to predict demand supply velocity for every ODVT and train multiple models to update prices without any manual inputs.

 

Note that our demand-side app is called RIVIGO Freight and supply-side app is called RIVIGO Fleet.

 

Tushar Makkar and Anshul Goyal from RIVIGO’s Data Science Team have also contributed to this article.