Making Transactions 'Fatter'

In the early days, when computer networks and processing power of point-of-sale and payment terminals was limited by then current-technology, the focus was on efficiency. Payment transactions generally only contained a few data elements of data required to process the transaction, and implemented technically in a manner that saved as many bytes as possible. This was important so all of this would work over a dial-up line and only send required data for transaction processing. Much of the payment message formats are tied to this legacy heritage to this date.

In the world of ‘Big Data’ there is a growing trend of providing fatter transactions, and providing more data in these transactions. These transactions consist of more then the final amount of the transaction and payment information, but now with market basket data and line item detail.

What does this involve from a payments system perspective ?

1) Expanding message formats and APIs to include list of skus and UPCs and other meta-data of market basket items.

2) Processing against a catalog to perform various value added services and processing.

3) Parallelism in transaction processing as certain items require processing that would take too long if processed in a serial manner.

4) Development of systems including robust engines and processing logic leveraging Machine Learning techniques to mine and process such data.

This isn’t new in concept as it has been performed locally in retailers for sometime now, as well as in some level-3 purchase/commercial cards. Now there is a trend of more value added services to enhance payment processing such as item based loyalty rewards, when such data is available you have more options and capabilities to enhance the payment transaction.

Idempotent Transactions

We recently were talking a lot about reversals this week in the OLS HQ, especially time-out reversals. Andy even mentioned his ever so famous “Refunds are not Reversals” So I was happy we were talking about reversals and not refunds ;)

Situation: What happens if you send a financial transaction to a payment system and we don’t get a response back? You are obligated to reverse it and keep on trying to reverse it (reversals are normally Store-and-Forwared (SAF) until you get a response back that the reversal was accepted.

You would be surprised how many implementations of payment software do not implement this important step, a disaster of not performing this is duplicate charges to cardholders during system or communication issues. This needs be be implemented in each path of a transaction. Terminal to Gateway, Gateway to Switch, Switch to EndPoint. for example. Many applications get-by, by ignoring reversals on Credit product types where cardholder have large open-to-buys and on Authorization Only Transaction Types. Reversal for Debit and and other financial transaction sets are a must.

On our Switch we handle Reversals with Idempotence. Wikipedia defines this as:

Idempotence ( /ˌaɪdɨmˈpoʊtəns/ eye-dəm-poh-təns) is the property of certain operations in mathematics and computer science, that they can be applied multiple times without changing the result beyond the initial application. The concept of idempotence arises in a number of places in abstract algebra (in particular, in the theory of projectors and closure operators) and functional programming (in which it is connected to the property of referential transparency).

Another website describes the problem as:

Problem: Network and server hardware failure can lead to lost messages, resulting in cases where a service consumer receives no response to its request. Attempts to reissue the request message can lead to unpredictable behavior within the service and the service consumer logic.

Solution : Design service capabilities capable of safely supporting repeated message exchanges.

Our implementation of reversals can handle multiple attempts of a reversal, we only process one but will accept any number of them. This is very important, Reversals are not “approved” or “declined” as the endpoint may or may not need to unwind anything. You as a caller don’t know whether the timeout was actually not processed at all, or if it was processed but you just didn’t get the response back.

We have the following ISO8583 v2003 based result codes in OLS.Switch for this so we can note the difference.

4000 Advice Accepted

4999 Advice Accepted - no Action Taken

That also means your logic is very simple - “send this reversal repeatedly on an interval until I get a response”

Q&A: Some questions on Payment Terminals

From the Mail Bag:

“I have a couple of questions you can help answer. Is it normal for a manufacturer to program POS terminals themselves? I have received contradicting answers to this question. Also, from the terminal, the [encrypted] cardholder information is sent to a processor. Do processors possess unique internet addresses that they give to the merchant to where the terminal can send this information?”

Great questions - let me take a stab at answering them:

“Is it normal for a manufacturer to program POS terminals themselves?”

It really depends - There are two models in play here, a) you can pay a terminal manufacture to development a terminal application, b) terminal manufactures also generally will sell SDKs, Software Development Kits, as well as required or optional training courses for independent developers to write payment applications for. From my personal experience, we have worked with both terminal manufactures as well as independent developers, as well as wrote very few in house.

“ Also, from the terminal, the [encrypted] cardholder information is sent to a processor. “

This is true in certain situations and depending on the application, terminal, and communication methods. Most dial terminals send cardholder information in the clear across a private dial line. Many IP/SSL terminals will just use SSL encryption as a transport mechanism for encryption/security. More recent generation of terminals and those that implement End-to-End Encryption (E2EE) or Point-to-Point Encryption will use both a data level and transport level encryption/security. Our message specifications and when we can enforce it, we always try to use tokenization, or surrogate numbers for subsequent transactions (Refunds, Captures, Voids, Reversals, - do not require the full PAN to be passed in many of our systems that we develop)

“Do processors possess unique internet addresses that they give to the merchant to where the terminal can send this information?””

Payment Processors and/or payment gateways will provide either dial 800 numbers for dial payment terminals or an IP address or https/SSL based URL for IP/SSL based terminals to send transaction data. OLS has integrates to various dial concentrator devices/networks - Hypercom NAC, TNS, HB.Net, now Phoenix Managed Networks, for Dial delivery. We have developed our own secure SSL Transaction Servers with various interface options for our customers, IP SSL Sockets, HTTPS Post, RESTful as well as SOAP based web services.

Q&A: What is the difference between a Cash Dispersement or a Cash WithDrawal Transaction

Q & A Time from the mailbag:

Scott Asks:

“In MasterCard, transactions Cash Disbursement and Withdrawal are regarded as two different transactions (They have different transaction type codes). What are the differences between these two transactions? What are their individual usage scenario ? “

Great question, A cash withdrawal is generally associated to a ATM or Debit Transaction where the full transaction amount can be immediately debited by the Card Issuer and Cash Provided to the Card Holder. While a Cash Disbursement transaction is generally something a Card Holder can request at a financial institution that he may not have a relationship with - Think of a cash advance of your Visa or MC card at a Bank, or if you have prepaid card or payroll card that you need to get money or cash from at a credit union that you don’t have an account with.

Transaction Types 101

Here is a list and definitions of common payment card transaction types:


The Authorization transaction is typically used by a merchant to obtain the authorization of a transaction amount as a pre-approval for the purchase of goods or services later during the fulfillment process. Authorization transactions are typically submitted for authorization and then funds are held by the issuer until that transaction is captured or the authorization is reversed or expires. An example can be found with online retailers who initiate an Authorization transaction to guaranteed funding by the card issuer prior to the shipment/delivery (i.e. fulfillment) of the goods. An “Authorization” is also referred to as an Auth-Only transaction.


A “Sale” transaction is used by merchants for the immediate purchase of goods or services. This transaction completes both the authorization and capture in a single transaction request. The Sale transaction is an Authorization and Capture transaction that if approved is automatically included for settlement.

Forced Sale

A “Forced Sale” is a transaction initiated by a merchant with the intent of forcing the posting of the transaction against the customer account without receiving prior authorization by the card issuer, or receiving a voice authorization code from the merchant acquiring call center. An example would be when a merchant’s terminal is offline, requiring the purchase of goods being completed without receiving online authorization by the card issuer. Or they received a Voice Approval. In these cases the merchant would enter the transaction details and forward this Forced Sale transaction to the card issuer with the expectation of receiving funding for the goods or services rendered. A forced sale does not require a matching authorization. Forced Sales are also known as Off-Line Sales.


A Refund allows a merchant to refund a previously settled transaction and submit the refund for processing. Refunds are only allowed for financial transactions (Sale and Captured) and are typically limited to the original authorization amount, or a lesser amount, in some cases, multiple partial refunds up to the original transaction amount. Some systems incorporate a feature called Matched Refunds. Matched Refunds must match back to an original transaction to help control fraud. “ Refunds” are also sometimes referred to as a “Credit” transaction.


Void transactions can reverse transactions that have been previously authorized or approved by the card issuer and are pending settlement. Merchants will only be allowed to void transactions that are in an open batch (pending settlement). Sale or Refund transactions are the most commonly voided transaction types.


The Capture transaction will allow merchants to capture a previously authorized transaction that is pending settlement, and submit it for clearing and settlement. An example is when online retailers who initiate an Authorization transaction to reserve funds by the card issuer prior to the shipment/delivery (i.e. fulfillment) of the goods, and then once fulfillment has been completed the transaction will be captured and submitted for settlement. A “Capture” is also referred to as a Pre-Authorization Completion transaction.

PIN Block Formats - The Basics

I had a call yesterday about approved TG-3 (Which is now called TR-39 ) ANSI PIN Block Formats. The TR-39 Audit Procedures state that ISO 9564–1 Format 0 (ISO-0) and Format 3 (ISO-3) are the only approved formats:

4.1.3 X9 Approved PIN Block Formats

Documented procedures exist and are followed that ensure any cleartext PIN-block format combined with a PIN encryption process has the characteristic that, for different accounts, encryption of the same PIN value under a given encryption key does not predictably produce the same encrypted result. (Note: any cleartext PIN block, formats 0 and 3 meet this requirement, as specified in X9.8-1).

Reference X9.8-1 - Sec. 4(c), Sec. 6.2, Sec. 8.3.1, Sec.8.3.2, and Sec. 8.3.5

In case you are curious here are Visa’s PIN Security Requirements

Requirement 3: For online interchange transactions, PINs are only encrypted using ISO 9564–1 PIN block formats 0, 1 or 3. Format 2 must be used for PINs that are submitted from the IC card reader to the IC card. Other ISO approved formats may be used.

This requirement further states:

PINs enciphered using ISO format 0 or ISO format 3 must not be translated into any other PIN block format other than ISO format 0 or ISO format 3. PINs enciphered using ISO format 1 may be translated into ISO format 0 or ISO format 3, but must not be translated back into ISO format 1.

(This last paragraph addresses an attack on Pin Blocks that can be translated in to format 1 on a HSM which would expose the clear PIN)

Let’s take a look at a few Pin Block formats:

For our examples: P - PIN Number F - Hex 0xF A- Last 12 digits of PAN not including check digit R - Random Hex Character (0-9, A-F) Let us use the account number 4111111111111111 and PIN Number 1234 (examples use a PIN Length of 4 but could be 4-12 digits)

“Pin Pad” format or IBM 3624

PPPP FFFF FFFF FFFF our Pin Block 1234 FFFF FFFF FFFF Notes: Not allowed and is an old legacy method - not approved to be used.


04PP PPFF FFFF FFFF (0 = ISO-0 Format, 4 = length of PIN) XOR with 0000 AAAA AAAA AAAA (Formatted PAN) our Pin Block: 0412 34FF FFFF FFFF XOR 0000 1111 1111 1111 = 0412 25EE EEEE EEEE Notes: Introduces variability in the PIN block by XOR’ing with a Formatted PAN - Best practice is to use ISO-3 instead of ISO-0 as there are attacks against ISO-0


1412 34RR RRRR RRRR (1 = ISO-0 Format, 4 = length of PIN) our Pin Block: 1412 348D 665A C5A3 Notes: Introduces variability in the PIN block by using Random padding chars - Best practice is not to allow HSM’s to accept or use this PIN Block format. Not allowed by TR-39 but is VISA.


34PP PPRR RRRR RRRR (3 = ISO-3 Format, 4 = length of PIN) XOR with 0000 AAAA AAAA AAAA (Formatted PAN) our Pin Block: 3412 34C8 CBA4 285C XOR 0000 1111 1111 1111 = 3412 25D9 dAB5 394D Notes: Introduces variability in the PIN block by using Random padding chars and by XOR’ing with a Formatted PAN - Best practice is to use this format.

10,000 TPS per second

I ran across Kiplinger’s article and picture of “The Credit or Debit Debate Visualized“ It is a very nice picture of both the usage of Credit and Debit Cards over time, as well as a nice list of pros and cons and differences between each, I encourage you to check it out for a good basic summary.

In payment systems I don’t really care as much about the type of card, Credit vs Debit at a basic level to me – one has PIN’s and requires usage of HSM’s and require real-time reversals and one uses clearing files and the other reconciliation files. But I digress.

At the bottom of the picture there are some statistics, Kiplinger being a personal finance website and not to mention various “Letters“, these are mostly consumer related, but his one caught my eye:


Which makes me chuckle because lots of prospects tell us they need a system to be able to support the worlds average TPS (Transaction Per Second), or a small fraction of.

Visa's Misuse of Authorization Fee


As of October 1st 2009 Visa has started to charge a fee of $.045 per misused authorization.

Visa defines a misused authorization as:

Authorizations that are not followed by a matching clearing transaction (or in the case of a cancelled or timed out authorization, not properly reversed)

For MOTO and e-commerce merchants who use payment gateways there may be some changes to how they perform Auth and Capture type of transactions, especially if they Authorize the transaction at order time, and later Capture or Complete the transaction when they ship a product. Or more typically, perform a $1.00 authorization first with AVS and CVV2 data, and followed by an authorization for the total amount of purchase.

Many payment gateways may or may have authorization reversals, authorization voids, or other transaction types - you will need to work with your gateway to identify what transaction types need to be performed to “reverse” an authorization for a product that you do not ship. Or if your practice is to re-auth after 10 days, and let the original expire - you will be hit with the fee.

Visa also supports a new transaction type called Account Verification - which is a $0.00 authorization - which they hope merchants will used instead of misused authorizations.

For processors and large merchants with direct connections to Visanet or Payment Gateways - These entities will want to verify that their reversal processing addresses credit reversals in the time-out scenario.

Now you're the Switch -- Successful Implementation Strategies

My colleague Andy Orrock wrote a blog post titled “Now, you’re the switch” where he summarizes challeges and witness poor implementions of some interfaces that we connect to:

But here’s the thing: once we send the transaction to you, now you’re the switch. What I mean by that is: your application is now beholden to the same throughput, speed, efficiency, extensibility and 24x7x365 availability concerns that define our lives. And while there have been many that have been up to the task, there have been countless other instances where that’s not been the case.

There are are few basic models that we have seen that work well:

  1. OLS Implemented Business Logic based on customer developed prototypes.

  2. OLS Implemented Business Logic and Customer provided Database or Flat File update feeds that drive authorization decisions.

  3. OLS Transaction Participants that call Customer Provided Software and CustomerSecretSauceManger.process() methods within our processing framework.

  4. Under our guidance, interface remotely to an local endpoint via TCP/IP Sockets or WebServices, handling concerns that we address here with tech savvy customers.

Pre-Authorization data for completions and reversals and removal of Track II Data

Late last week I received a email detailing a few message format specification changes for a processing gateway interface that OLS.Switch connects to. It discussed changes to the required data elements required for “match-up”, the data required in the original transaction that one must return in a reversal response message or in completion messages. We leverage Host Reversals (note: these are not the same as refunds) when we don’t recieve a response from an authorizer for remove authorization, most typically on a time-out scenarios. We don’t know if the processor accepted the transaction and we just didn’t receive the response or if the processor never received the original transaction at all. In cases like these we are obligated to send reversal messages, to reverse the transaction. In a credit world where there are large open-to-buys and credit limits and expiring authorizations, this is less of a deal with debit. In the debit world mistakes here case pain for cardholders and duplicate charges occur. We SAF (Store and Forward) our reversals and retry for a set period of time with delay intervals until we receive an acknowledgment that our reversal transaction was received. Completions can be forced sales or post authorization transactions, or in certain industries completions with updated amounts that differ then the authorization (Think restaurant tip and gas pumps transactions here.)

The changes are listed below:

Track 2 data is no longer required for completion, reversal or void transaction types. With the elimination of the Track 2 Data (Field ID 35) these transaction types will now require the Account Number (Field ID 02) and Expiration Date (Field ID 14) to be provided. All clients should make these modifications prior to your next PCI assessment.

This is great news, the is one of the last processing gateways that we interface with that required the ISO8583 Data Element 35 - Track 2 Data for reversals and completions. If you ever noted the specific wording in the PCI DSS specification about “subsequent to the authorization” this is part of why I believe that wording was left there. The issue with this is while SAF queues are normally located in memory - there are times when they can be configured to be persisted to disk (If they are not, think of the cardholder impact of duplicate charges) This is less data, specifically pre-authorization data that is required to be stored prior to the authorization. We have leveraged “encrypted spaces” to help protect all types of SAF Queues and are happy Track 2 Data is not required for match-ups any more - Ideally, and we hope that processors will take this a step further and remove the requirement of sending the PAN, and leverage other unique transaction identifiers or composite identifiers: Take a look at one our our “FindOriginal” Transaction Participants for a MasterCard Interface:

CriteriaImpl(TranLog:this[][date>=Thu Sep 17 09:11:41 CDT 2009, irc=1816, stan=000000087625, originalItc=100.00, acquirer=987654999, mid=123456789012345, tid=12234501, banknetReference=MQWWRJ4QW ])

No Card Number there !

Your browser is out-of-date!

Update your browser to view this website correctly. Update my browser now