Product Sync Status

Objective

As an admin, I want to be able to give Sellers concrete information in case the product synchronization lasts a long time. Therefore, it is necessary to know the product’s status during the entire synchronization process to the Shop system.

Sync Flows

Currently, there are two flows in place to sync products from Seller Center to the Shop backend system. All product-related calls are handled the same way (either synchronously or asynchronously). There is no configuration in place that allows only one flow to be used for certain product calls.

Synchronous Flow

Seller Center uses this synchronization process by default. Product request are sent based on this documentation.

Asynchronous Flow

For the asynchronous flow, the same requests are sent to the Shop backend system, but they are handled differently there.

  • The final response regarding whether or not a product is synced is sent to Seller Center at a later time.
  • During this time gap, Seller Center is able to send further requests and the Shop system can process the already-received requests.
    Critical KnowledgeIf you want to enable the Asynchronous Sync Process (Shop update), please raise a TMLSD ticket. The parameter that needs to be enabled is shop_update/ flow. When this configuration gets updated the actural change in the flow will take up to 1 hour to be activated as internally the cronjob which handles this synchronisation is not yet updated. Either for up to 1 hour both flows need to be supported by the Shop system or on the server the cronjobs need to be restarted.

    Furthermore please provide the following information: IPs which need to be whitelisted. Also included in the ticket. API-key for authentication, shared via email to [email protected].

    For GET: The system credential which needs to be updated is bob_callback_server.

    For more technical information, please read here the backend documentation and the according API-documentation.

     

     

    Synchronization Overview

    An overview of all items that are in the queue and the performance of the sync process is available under Administration > Shop Update

    Pending / Processing Tab

    This tab lists all products that are currently in the queue to be synced with the Shop system. Especially with the asynchronous approach, it is necessary to check whether or not the request has already been sent to the Shop.

    “Seller SKU”: SKU of the product that has been provided by the Seller

    “SKU”: SKU of the product that has been generated on Shop-side

    “Topic”: Type of synchronization

    • “Content”: Content changes for a product
    • “Images”: Image changes for a product
    • “New Product (Set)”: New product set (multiple variations for the same product) created (on Shop-side, this is often called “config”)
    • “Price”: Price changes for a product
    • “Stock”: Stock changes for a product
    • “Status”: Status changes for a product (for more information, please see below)
    • “Consignment Stock Change”: Consignment stock information for a product
    • “QC Status Change”: Approval status change for a product
    • “New Product”: Adds a product to an existing product set (config) as a new variation.

    “Queued at”: Time when the product was added to the queue

    “Status”: Sync status of the product (if the Shop system has already acknowledged it).

    “Requeued count”: How many times Seller Center has tried to resend the product

    “Last requeued at”: Time when the last request was sent to the Shop system

    		 sync_queue2

    Status Explanation

    Status Explanation
    “Pending”
    • Queue item has just been persisted.
    • There is no request-payload yet.
    • Items should not remain in this status for more than a few milliseconds.
      • An item will only stay in this status longer if there is a specific reason blocking it.

    If such a case is encountered, please create a TMLSD ticket.
    “Processing”
    • Status between pending and Building (Request) Payload
    • The item has been picked to be processed
    • Item should remain in this status for, at most, a few seconds
    “Building (Request) payload” The request-payload is built and persisted in Seller Center.
    “Made request” Status of item before the request is sent to the Shop system.
    “Got acknowledgment”
    • Only available in asynchronous flow.
    • The items that are sent to the Shop system are queued in their own queuing system. The Shop systems sends an acknowledgment back to indicate that everything is fine.
    • If an item is stuck in this status, it means that the Shop system has not sent a callback with a response.
    “Got response” A response has been received from the Shop system that the item is also queued on the Shop-side.
    “Building response payload” The payload has been extracted from the response.
    “Response processing”
    • The post-update processing (e.g., update catalog_product.src_updated_at) is about to be performed.
    • Product has been synced with the Shop system
    “Delayed”
    •  Queue item processing failed (for any reason) and is delayed. It can be retried later.
    • This is a product\stockchange update and Seller Center has to wait 10 minutes (for order processing delay reasons) until the stock change is synchronized to the Shop backend system.
    “Blocked”

    Currently, all items are blocked if the product\productset does not have src_id/shop_sku yet, or has not yet been synchronized to the Shop system .

    This means:

    • The product is new and has not yet been created in the Shop.
    • If a Seller updates their products (e.g., price, stock, status), those changes are immediately pushed to “Blocked” because the product has not yet been created in the Shop backend.
      • Those changes are saved until the product is created.
      • Afterwards, the changes are requeued to be synchronized.
    • If a product already exists in the Shop system, but the Seller adds new variations to the product, the synced product reach the blocked states until the new variations also have a Shop sku
    • Synced products are in blocked state, if Seller Center receives from the Shop a response, which can’t be processed, like an error message or wrong data.

    Delayed

    This tab shows all products that are delayed for synchronization. Products are delayed when Seller Center receives a failure from the Shop system.

    Failures can include:

    • Error 404, 500 from the Shop system
    • Correct response, but unsuccessful response from the Shop
    • Any exception errors in the synchronization (Shop update)

    “Seller SKU”: SKU of the product that has been provided by the Seller

    “SKU”: SKU of the product that has been generated on Shop-side

    “Topic”: Type of sync

    • “Content”: Content changes for a product
    • “Images”: Image changes for a product
    • “New Product (Set)”: New product set (multiple variations for the same product) created
      • On Shop-side, this is often called “config”
    • “Price”: Price changes for a product
    • “Stock”: Stock changes for a product
    • “Status”: Status changes for a product
    • “Consignment Stock Change”: Consignment stock information for a product
    • “QC Status Change”: Approval status change for a product
    • “New Product”: Adds a product to an existing product set (config) as a new variation

    “Queued at”: Time when the product was added to the queue

    “Status”: Sync status of the product (whether the Shop system has already acknowledged it)

    “Requeued Count”: How many times Seller Center has tried to resend the product

    “Last Requeued at”: Time when the last request was sent to the Shop system

    “Retry Logic”: Seller Center uses the leaky bucket algorithm to requeue a failed item. The algorithm uses the formula MIN(Retry³ (calculated in minutes), 8 hours), which guarantees that Seller Center performs at least three retries per day. Seller Center retries each item 27 times in total. The retry table below explains the timing:

    Retry Minutes Additional Delay in Minutes Total Delay in Minutes
    #1 1 00:00:01 00:00:01
    #2 8 00:00:08 00:00:09
    #3 27 00:00:27 00:00:36
    #4 64 00:01:04 00:01:40
    #5 125 00:02:05 00:03:45
    #6 216 00:03:36 00:07:21
    #7 343 00:05:43 00:13:04
    #8 480 00:08:00 00:21:04
    #9 480 00:08:00 01:05:04
    #10 480 00:08:00 01:13:04
    #11 480 00:08:00 01:21:04
    #12 480 00:08:00 02:05:04
    #13 480 00:08:00 02:13:04
    #14 480 00:08:00 02:21:04
    #15 480 00:08:00 03:05:04
    #16 480 00:08:00 03:13:04
     #17 480 00:08:00 03:21:04
     #18 480 00:08:00 04:05:04
     #19 480 00:08:00 04:13:04
     #20 480 00:08:00 04:21:04
     #21 480 00:08:00 05:05:04
     #22 480 00:08:00 05:13:04
     #23 480 00:08:00 05:21:04
     #24 480 00:08:00 06:05:04
     #25 480 00:08:00 06:13:04
     #26 480 00:08:00 06:21:04
     #27 480 00:08:00 07:05:04
    		shop_update_delayed

    Failed

    This tab shows all products that are delayed for synchronization. Products are considered “Failed” when Seller Center receives a failure from the Shop system 12 times.

    Failures mean:

    • Error 404, 500 from the Shop system
    • Correct response, but unsuccessful response from the Shop
    • Any exception errors in the synchronization (Shop update)

    “Seller SKU”: SKU of the product that has been provided by the Seller

    “SKU”: SKU of the product that has been generated on Shop-side

    “Topic”: Type of sync

    • “Content”: Content changes for a product
    • “Images”: Image changes for a product
    • “New Product (Set)”: New product set (multiple variations for the same product) created
      • On Shop-side, this is often called “config”
    • “Price”: Price changes for a product
    • “Stock”: Stock changes for a product
    • “Status”: Status changes for a product
    • “Consignment Stock Change”: Consignment stock information for a product
    • “QC Status Change”: Approval status change for a product
    • “New Product”: Adds a product to an existing product set (config) as a new variation.

    “Queued at”: Time when the product was added to the queue

    “Failed at”: Time of the last attempt to send the product to the Shop system.

    		 shop_update_failed

    Re-queuing Failed Product Calls

    If a call is in the “Failed” status, it may be possible to re-queue the item with up-to-date information.  This action is not currently available in the Global Product Overview page. This does not re-queue image related items.

    # Description Image
    1
    • Go to Products > Manage Products
    2
    • Click “Edit,” then “Reupload to Shop”
    		dispose_request
    3
    • A confirmation dialogue will be displayed to remind the user that the action will affect all variations in the product set.
    • Click “OK” to confirm.
    		return_request
    4
    • The item is now re-queued.
    Critical Knowledge Items can only be re-queued if they meet the following requirements:

    • The Seller is active.
    • The product is active.
    • The product is approved in quality control.

    This action is only available for admin users.

     

    Analytics / Reports

    In this tab, different graphs are displayed to illustrate the volume of synced products.

    Sync KPI

    This graph shows the amount of synchronization requests that have been sent to the Shop system. The different colored graphs show the percentile of the queue that have been successfully synced within the last 24 hours, 7 days, 14 days, 30 days, and 90 days.

    Queue Size

    These graphs shows the size of API calls, split by type for the selected time. Their sizes indicate how many products have been sent to the Shop system. The message on top of the page displays the amount of products that need to be sent to the Shop system or are waiting for acknowledgement. A high number of “blocked” products simply means that these products have updates in the queue, but they have not yet been created in the Shop system. Once they are created, the remaining product updates are sent.

    API Calls

    These graphs shows the amount of API calls, split by type for the selected time. Furthermore, a split view of successful and failed calls is displayed.

    API Average Response Time

    These graphs shows the size of API calls, split by type for the selected time. Time refers to milliseconds.

    Kibana Usage

    Kibana is a tool that enables Rocket Labs to monitor and investigate the Seller Center logs that are generated on the server. Via elastic search, it is possible to search huge amount of data (e.g., when checking information on product data was synced to the Shop system). Currently, log files from the last three month can be searched within Kibana.

    In order to access the Kibana logs, please contact the Rocket Labs Service Desk so that the according rights can be granted to the user’s JIRA account. It is mandatory that a JIRA account is created beforehand. Please send an email to [email protected] and request the user creation.
    Description Image
    Go to Kibana.
    Overview explanation:

    1. Collapse or expand the time range area to select either a relative time or a specific time frame.
    2. Select the date range to set an absolute time in which log files should be searched.
    3. Enter the search query.
    4. Amount of successful hits where the search query has been found is displayed. This graph can be used to narrow down the results within a specific time frame. Graph is displayed relative to the searched time.
    5. Collapsible results of the log entry are displayed.

    General Kibana Information

    Description Sample Search Query
         		Image
    Searching logs for a specific Venture on specific environment 
    venture:“venture name” AND source:“country short code/environment”
    • Venture name: dafiti, daraz, iconic, jumia, kaymu, lamoda, linio, lyke, wadi, zalora,
    • Country short code: e.g. id, mx, pk, hk, th, ru, eg, …
    • Environment: live / staging

    Note: Do not use an asterisk in the beginning of the search query when searching for a specific Venture or specific countries.
    Search for a product SKU (Seller SKU or Shop SKU)  

    venture:“zalora” AND source:“eg/live” AND message:“SKU420202”

    Displays all entries with this SKU. The search results will also include searches on Product or Order Overview for this particular SKU.

    Click the small arrow on the left to open each result.

    • Here you can see the full API call and further details.
    Search for product create call 
    venture:“zalora” AND source:“eg/live” AND message:“SKU420202” AND message:“Product.createProducts”

    The request contains a request ID at the end. This ID is needed in order to  find the response to this specific API call later. This works the same for any product call (e.g., product create, update, stock update, or price update).

    Click the small arrow on the left to open each result.

    • Here you can see the full API call and further details.
    Search for a product response from the Shop system 
    venture:“zalora” AND source:“id/live” AND message:“5806653” (AND message:“RawResponse”)

    Include the request ID from the request in the search query. You can also add the section with the RawResponse to have a unique result if the ID is also being used in a different context (e.g., order [item] numbers).

    Make sure that if the Venture uses the synchronous flow, the same time frame is used; this makes the result distinct. When using the asynchronous flow, search for the request ID and the section for RawResponse within a more open time range (the response can be sent to Seller Center a few hours, or even days, later).

    Click the small arrow on the left to open each result.

    • Here you can see the full API call and further details.
    Order-related searching 
    venture:“zalora” AND source:“id/live” AND message:“45123” AND message:“order_call”

    The best way to search for order-related logs is by using the OMS order item ID and the API call (e.g., getMarketplaceOrders()-call or any order status update call).

    When there are multiple products within one package, Seller Center sends all order item IDs to OMS only for package creation. Otherwise, Seller Center only uses one randomly-selected ID for further processing calls.

    Click the small arrow on the left to open each found result.

    • Here you can see the full API call and further details.

    Sharing Kibana Links

    In order to share Kibana links, please use the share functionality within Kibana, as opposed to directly copying the link from the browser address bar.

    This makes it easier to see the searched results. Otherwise, you risk having to restart the search with the same criteria.

    Once you click “Share,” a panel expands and displays the link that can be used.

    As the link can be very long, there is also the option to create a short URL that encodes the link and reduces its length. This new URL can easily be copied to other locations (e.g., to TMLSD tickets).

     

    Asynchronous ShopUpdate flow FAQ

    +-Product can’t be updated although successfull created on Shop side

    It needs to be checked, which product id was used from the Shop System, when the product got created. It can happen, that on Seller Center side this id is already assigned to a different product and therefore the first one can’t be updated.

    +-

     

     

    +-

     

     

    +-

     

     

    +-