Put ‘request’, ‘response’ tranlog columns in new table

My partner in crime, Andy Orrock, writes a post about a feature (more of an enhancement) that we have implemented in our OLS.Switch product in a recent blog post titled: Put ‘request’, ‘response’ tranlog columns in new table, I wanted to add some of my own commentary on this change. Please read Andy’s Post first before continuing.

As a Payment Switch there are times ( especially in development / testing ) that you will want to log or see what the switch is sent from a terminal or POS system, or sent and recieved from an authorization end-point. This feature is very handy during integration to new end-points, different message formats, changes with additional data elements and initial testing and certification efforts in test environments. In Production this is very, very bad, because raw messages contain card-numbers, Track Data, CVV2 Data, PIN Blocks, and all of the other “Bad” stuff one is prohibited to store according to PCI. OLS.Switch by default has this feature turned off, and recommends its use as a last resort for troubleshooting production problems.

Let me rip the introduction paragraph and a few bullets from our PABP Implementation Guide:

Secure Troubleshooting Procedures

OLS.Switch is configured to use various techniques to either protect or wipe sensitive cardholder and authentication data to prevent storage of prohibited data, or to use encryption to render the card number unreadable.

There may be instances in which sensitive cardholder information or sensitive authentication data needs to be viewed for troubleshooting purposes. Sensitive authentication information must only by collected when needed to solve a specific problem. The following are secure troubleshooting procedures designed to allow limited controlled access for troubleshooting purposes, all steps must be followed. You must be authorized and approved to make these system configuration changes. Furthermore, it is recommended that your internal company’s Change Management and Problem Management policies and procedures are followed in conjunction with these procedures.

<!-- /* Font Definitions / @font-face {font-family:Arial; panose-1:2 11 6 4 2 2 2 2 2 4; mso-font-charset:0; mso-generic-font-family:auto; mso-font-pitch:variable; mso-font-signature:3 0 0 0 1 0;} / Style Definitions / p.MsoNormal, li.MsoNormal, div.MsoNormal {mso-style-parent:””; margin:0in; margin-bottom:.0001pt; mso-pagination:widow-orphan; font-size:10.0pt; mso-bidi-font-size:12.0pt; font-family:”Times New Roman”; mso-ascii-font-family:Arial; mso-fareast-font-family:”Times New Roman”; mso-hansi-font-family:Arial; mso-bidi-font-family:”Times New Roman”;} @page Section1 {size:8.5in 11.0in; margin:1.0in 1.25in 1.0in 1.25in; mso-header-margin:.5in; mso-footer-margin:.5in; mso-paper-source:0;} div.Section1 {page:Section1;} / List Definitions */ @list l0 {mso-list-id:1140000935; mso-list-type:hybrid; mso-list-template-ids:-872906204 67698703 67698713 67698715 67698703 67698713 67698715 67698703 67698713 67698715;} @list l0:level1 {mso-level-tab-stop:none; mso-level-number-position:left; text-indent:-.25in;} ol {margin-bottom:0in;} ul {margin-bottom:0in;} –>

x. Determine if troubleshooting can be performed on test environment with test card numbers. Perform troubleshooting in that environment first.


<!-- /* Font Definitions / @font-face {font-family:Arial; panose-1:2 11 6 4 2 2 2 2 2 4; mso-font-charset:0; mso-generic-font-family:auto; mso-font-pitch:variable; mso-font-signature:3 0 0 0 1 0;} / Style Definitions / p.MsoNormal, li.MsoNormal, div.MsoNormal {mso-style-parent:””; margin:0in; margin-bottom:.0001pt; mso-pagination:widow-orphan; font-size:10.0pt; mso-bidi-font-size:12.0pt; font-family:”Times New Roman”; mso-ascii-font-family:Arial; mso-fareast-font-family:”Times New Roman”; mso-hansi-font-family:Arial; mso-bidi-font-family:”Times New Roman”;} @page Section1 {size:8.5in 11.0in; margin:1.0in 1.25in 1.0in 1.25in; mso-header-margin:.5in; mso-footer-margin:.5in; mso-paper-source:0;} div.Section1 {page:Section1;} / List Definitions */ @list l0 {mso-list-id:1140000935; mso-list-type:hybrid; mso-list-template-ids:-872906204 67698703 67698713 67698715 67698703 67698713 67698715 67698703 67698713 67698715;} @list l0:level1 {mso-level-tab-stop:none; mso-level-number-position:left; text-indent:-.25in;} ol {margin-bottom:0in;} ul {margin-bottom:0in;} –> x. Only collect the limited amount of information needed to solve the specific problem. Only collect enough data in the troubleshooting log that is required to address the specific problem

(There are lots of other steps and controls to verify that any changes are set back to default, appropriate destruction of captured data is handled, etc, etc, etc.)

Logging raw messages is a dangerous feature and much care needs to be taken with it, and is rightfully heavily scrutinized with knowledgeable PCI Auditors, while not an issue in a test environment using test card numbers, a system misconfiguration or human mistake or “forgotten” changed setting in production could prove disastrous. OLS has added some “controls” around this feature.
Previously there were columns in our TranLog that were called REQUEST and RESPONSE, in order to enable this type of logging an entity such as a store or specific terminal (Terminal ID) would need to be configured and enabled to do so, and would need to follow all of our “user controls” and recommended procedures (including preventive and detective controls) in our PABP guide. For the record non of our clients on our production system have any data in the REQUEST and RESPONSE columns of the TranLog in production environments. I’m happy that it is not a widely used feature in production.

With the new release we now have a single related table called raw_request that has a relationship with a transaction in the TranLog - a much cleaner and normailzed approach. In addition to this, there is a system-wide parameter called auditTrace for each OLS.Switch module that must be enabled by setting the value to true, it is defaulted to false. These system wide parameters are based off of configuration files, and we recommend that our clients use File Integrity Monitoring to detect and alert on any changes to application configuration files. Once the system-wide parameter for the modules are enabled, a specific store or terminal needs to be configured and enabled; It is a two step process. In addition, This approch also makes it easier to detect if the system is configured in a “non-compliant” fashion - we have monitoring tasks and alerts that scan the raw_message table, and alerts if the row count is non-zero. Also if there is any Database replication or archiving, moving this data to a separate table, ensures that troubleshooting data remains and isn’t disseminated.

This feature is a necessary evil that most of our customers ask for or have in other Payment Switches (we do have the ability to remove the raw_message table and functionality completely). We hope that further adding preventive controls (Making it harder to enable, user controls to use dual control and have secure troubleshooting policies and detailed secure troubleshooting procedures to follow), detective controls (user controls to detect application configuration changes and monitor row counts of the raw_message table) ensure that it is an intentional change on the customer’s part to enable this functionality.

Also: the following paragraph by Andy shows off our different biases:

One follow-up to this Release Note: I asked Dave how we should set ‘auditTrace’ in production – my thought was to set it to ‘true,’ thinking we’d be at the ready to turn on tracing without a service re-cycle. Dave strongly disagreed and stated: OLS.Switch ought to be “Secure by Default” in production. I really liked that.

Dave = Security Focused.
Andy = Operations and Timely Troubleshooting.

Payment Systems / Application Demos and Presentation thoughts

Photo by The Eggplant

Over the last few months there has been various webEx, gotoMeeting, Live Meeting, etc of product demonstrations that I’ve been a part of as a participant.

Some General Thoughts:

  • If you are showing a web based application use a SSL Certificate and https:// If you are going to show a web-interface that you log in with a username and password or shows account numbers please do this- you can used a self-signed cert, but I get nervous about demo’s without this- It is just sloppy not to do.
  • Mask Account Numbers when they are displayed.I get really nervous about this type of stuff and question your security posture.
  • Don’t use account numbers and PIN as authentication method, (although there are certain instances where this is acceptable) don’t make this the default option.
  • If you are showing a payment system - understand what PABP and PA-DSS are - and if you have customers that are “PCI Complaint” running it, this isn’t the same to me.
  • Show a finished product, links that go to “Not yet completed” or pages that are not consistent in look and feel confuse me.
  • When I ask how many ‘Live Customers’ use this product, I want to know about in production, not in the sales pipeline.
  • If it is a MS Windows/SQL Server based product, don’t list Windows Std. Edition and MSSQL Standard Edition as required software - We need enterprise level software, there is a huge delta in TCO in licensing fees.

Things to do right:

  • Simulate a live transaction against a simulator or other tool showing that it is a real system and is functional.
  • Walk me through the life-cycle of certain processes that I care about.
  • Be able to explain “how you would implement X” or modify Y, or how your system deals with “Z”
  • I know that a product won’t solve all of my needs, so I’m looking for synergies with your team to be partners with a relationship to get your product to fit my needs.
  • Be able to speak my language, and have a few competent people driving the demo.
  • Show me how “someone would use this” application in the real world.

Encryption options from the POS/Payment Device/Terminal


There are a few different ways of implementing “encryption” from the POS/Payment Device/Terminal, I though I’d look at a few in a short post:

1) Tunneling - using existing applications and their connections over an encrypted tunnel e.g.over a VPN, SSH, stunnel, etc. This approach doesn’t require any changes to devices or message formats or the payment “server”

2) Transport level - using TLS/SSL over TCP/IP Sockets - or at a higher level (web server/web service) using HTTPS. - devices needs to support the ability and make this type of connection, message formats are not modified.

3) Data Element or Field level – if you only want to encrypt the PAN or other specific fields, and these fields are defined to support the increased length required of the encrypted payload – this requires changes to the message formats, devices and payment “server” software. Consider truncating the Account Number/Track Data in DE 2 or DE 35 in ISO8583 for use of displaying purposes on the terminals screen or receipt and consider using another Private ISO field for the payload.

The approach will depending on what the “devices” sending the transaction can support, both from a connection perspective as well as a software perspective. I’d also recommend consider using asymmetric encryption rather than symmetric here, as then the devices would not have the ability to “decrypt” as they would not have the private/secret key, and would help with eliminating private key storage at the device level if you choose option 3. There are implementations that use HSM’s and the DUKPUT algorithm as well.

We have an implementation of #3 that I wrote about here. -- relevant paragraph below:

Some of our implementations of OLS.Switch supports field or data element level encryption that is passed on from the Point of Sale system to our switch. The main thing that allow us to perform this is that: We or our customer “own/control” the POS message format to us and can adapt and handle the programming of the POS System and POS message formats - our account number fields are not limited to 16 digits - we can handle a much large encrypted value. So over the wire - these implementations are “protected” from eavesdropping or sniffing.

I plan to write more on E2EE (End to End Encryption) in the coming weeks as well, so stay tuned !

Operations Considerations Batch Files and Extract and Import jobs


I’ve been working some batch file based jobs for a project here for OLS. There are two sides of this; sending “clearing” files of transactions in a defined format to third party which I will call the extract, and receiving “refresh” or table updates from this third party, I’ll refer to this as the import. The extract file contains financial transaction records, and the import file contains entity information such as merchant information. The Extract File Layout is quite standard an looks something like:

File Header

—-Merchant Header

——–Batch Header

————Detail and Addendum Record(s)

————Detail and Addendum Record(s)

——–Batch Trailer

—-Merchant Trailer

File Trailer

The File Layout for Import is a list of fields per record, nothing real fancy here.

I don’t want to spend too much time about the actual mechanics of the Extract and Import jobs themselves, but rather the Operational Considerations of this and others that we have performed:

Validity – You need to decide how to handle invalid records in a file or valid records without proper supporting data (transactions for a merchant that wasn’t setup in your system) ? You can write off the bad record to an exception file, and address it later, or you can reject the full file, the approach depends by implementation and requirements. We also mark files with a .bad extension if we detect they are invalid to help prevent subsequent processing steps, like transmitting a half-baked file. We also perform duplicate file checking as well as other validation steps.

Completeness – You need to make sure that you read in the entire file or extract a complete file. Monitoring controls such as checking the number of lines and file size in an Extract file, as well as checking the last time of a file for a specific record such as a File Trailer. Reconciliation between hash totals and amounts is also a good practice. On the import side you can count the number of lines or records in a totals from a trailer record and compare that to what was imported.

Timeliness – Some extracts take minutes and others hours, scheduling and monitoring the process is essential to perform data processing on a timely basis to other parties. Monitoring “check-points” in the process as well as a % of records completed help here to detect problems proactively with monitoring. Collect job performance metrics, it is valuable to keep track of and chart the total run time of each job and compare it to its history, to detect slowdown’s or to correlate increase or decrease in process times with external events or transaction growth.

Delivery – Consideration for the delivery of the file must be made. File Transfer procedures that address file naming conventions, steps to upload a complete file (upload as a .filepart extension and the rename to the full name upon complete transfer) as well as secure delivery as well as archiving locally or remotely, compression, and any file level encryption. It is also a good practice to reconnect to the file server and perform a directory listing on the files that you uploaded to confirm that they were transferred successfully.

Security – While account numbers and such are encrypted in databases (column level, file level, internal database level) the file specifications don’t allow for encrypted card-numbers, so both file level asymmetric encryption using the public key of the recipient as well as transport level encryption to send the file (see Delivery above) need to be considered. Archival files stored on disk will also need to be stored encrypted as well.

Troubleshooting/Restart Procedures – You need to develop procedures to support the following:

· re-sending failed files

· re-running the extract or import process for a specific date

· preventing duplication or invalid files or data.

The End is Just the Beginning – Operations is just the start of a process that has no end, it requires daily care and maintenance. These processes and controls need to work in harmony on a continuous basis and be able to be enhanced based upon the results of monitoring and other operational tasks.

Steps to setup a successful endpoint integration


We connect and interface to about 20 different endpoints, each has its own message formats, connection methodologies, protocols, and different implementations of payment transactions sets, encryption and key management considerations, On-Line vs Offline transactions, Store and Forward (SAF), and flat file based reconciliation and clearing files among other details.

Here is a basic list of steps that we perform that assist us in getting the endpoint integration right:

1) Acquire the Host Message Format specifications and communication/protocol specifications that describe the communication and connection e.g: 2||4 byte length header, other header(s), length in network or host byte order, etc. and read and understand them. Perform this same task for batch flat file based formats and file transfers.

2) Request sample dumps/files of some messages as well as for online transactions, network traces that show the communication bytes of the message, manually parse them and map them back to the spec. These should include samples of request/response exchanges. [These should be from a test system.] Examples often reveal nuances not documented or kept up to date in the spec.

3) Determine which message set and transactions are in play, There are many types of transactions that will be different depending on the type (e.g Debit, Credit, Gift) and characteristics of the transaction (e.g Manual, Swiped, MOTO). You need to determine which ones map to your business requirements and which ones will be used.

4) Request and/or develop certification scripts for your implementation (it helps to see what the other side thinks is important).

5) Schedule a walkthrough – This is the key event: the probability of our success is increased by scheduling and holding one or more walkthroughs. The idea is to comb through the spec(s) and complementary material page by page, field by field and get complete alignment on the task. It’s the absolute truth that the real nature of the interface comes out in the walkthrough.

6) Develop and test your code to the specs and sample messages dumps. (Which is a small part of the project)

7) Use tools such as netcat and wireshark as well as your application logs to confirm that you are sending what you think you are, and can provide detailed information to someone on the ‘other’ end of the endpoint to help troubleshoot integration. Don’t assume that you think you are sending what your think your code does, verify it locally first.

8) Develop a good or rather great working relationships with someone on the ‘other’ end of the endpoint, that can help assist with message format questions, perform traces, and check what you are sending to what they expect, and setup regular communication throughout the project.

Payment Systems Tools - Netcat

Netcat is described as the Swiss Army Knife for TCP/IP. I use it in two different ways:

1) Build a small server that dumps the contents it receives to a file. (Helpful for reverse engineering and debugging purposes)

$ nc -l -p 10000 > out.txt

This will set netcat to listen on port 10000 and redirect the output to the file out.txt

For Instance, I configured a client program that sends messages to a server and connected to my localhost:10000 instead:

$hd out.txt

00000000 00 4d 30 31 30 30 70 2000 00 00 c0 00 00 31 36 |.M0100p ……16|

00000010 34 31 31 31 31 31 31 31 31 31 31 31 31 31 31 31 |4111111111111111|

00000020 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 31 |0000000000000001|

00000030 30 30 31 32 33 34 35 36 31 32 33 34 35 36 30 31 |0012345612345601|

00000040 31 32 33 34 35 36 30 31 30 31 30 31 30 31 20 |12345601010101 |

This shows me that there is a 2 byte header (00 4d) of 77 bytes in front of this ISO8583 0100 (30 31 30 30) message.

2) Use it as a client to send message dumps. (Helpful for a quick and dirty method to send data over TCP/IP)

$ cat out.txt | nc localhost 10000

This will send the contents of out.txt and ‘pipe’ it to netcat that will attempt to connect to localhost on port 10000 and send the contents of out.txt.

SPSP Webinar: Secure Hashing and Scope Reduction Methods

Hashing is one of the most basic constructs of Payment Systems development, as it relates to both secure storage of passwords as well as a method of rendering card numbers unreadable (PCI 3.4), you can also use the hash as a lookup or index in a database if you are performing column level encryption especially asymmetric encryption. I hope the presentation includes the difference between “secure hashing” and “naked hashing”

I encourage my readers to attend the following webcast by the Security of Payment Security Professionals, of which I am a member.

Wednesday, January 21th, 2009 the Society of Payment Security Professionals is hosting a webinar on “Secure Hashing and Scope Reduction Methods”. You can register for this event online.

I plan to do a follow-up post on this and perhaps with a few code examples :)

jPOS Twitter Integration

For fun I decided to develop a TwitterLogListener class for the jPOS Framework. It leverages JTwitter - the Java library for the Twitter API from Daniel Winterstein. With this, you can send alerts and/or messages to a twitter account for monitoring from a jPOS application. You need to configure a TwitterLogListener in your logger:

And you need to Log an event with a ‘twitter’ tag:

Look at the messages in a twitter account: http://twitter.com/paymentsystems :) For more details on this see this post: http://is.gd/eDbI

Batch vs. Real-Time Processing in e-commerce environments

I was having a discussion recently about payment processing in e-commerce environments, specifically Batch versus Real-Time Authorizations. Batch processing has a one-a-day connotation, or that it is simply file based. Why would you not want to send transactions real-time was the question to me?

My Answer: Any time you are not face to face with a customer or if there is a delay before the customer receives the product or service, you should do Near-Time authorizations with the intent of not allowing the mechanics of the authorization impact the customer.

Let me explain:

It comes down to the customer checkout experience as well as to help prevent shopping cart abandonment the payment process should not be an impediment to the customer ordering an item. Especially if there is a glitch, communication issue, or any other outage in the authorization process including late or slow replies. You don’t want your customer to get frustrated during the order process, because they will go some where else to shop.

What is an alterative approach ?

Take the order, log it in a Database with a status of “Payment Pending” or “Authorization Pending” Have a batch process or scheduled task or cron job that runs on a periodic basis to loop through the list of Orders with Pending Payment Statuses, Securely acquire the required authorization data elements required for the authorization request (Considerations need to be made regarding the retention of CVV2 subsequent to the authorization request, and how the authorization data is managed between the order and authorization request ) and perform Near-Time Authorizations for these orders. Approval responses update the Order Records in the Database, and “errors” and are placed to try again at the next run (Consider Reversals and Duplicate Transaction Checking here on error responses to protect your customers open-to-buy) Declines are noted as Payment Failed, and you send a payment email allowing your customer be notified that the authorization failed, and prompt them to update their payment information on a Secure Account Web Page or call you to provide it over the phone. (Note this method wont work if you are doing any 3D-Secure - Verified by Visa, MC Secure Code , stuff – Which I’ve used I think 2 times in the last 5 years.)

Our you could do it real time and put a message on your cart to your customer, “Please Try Again Later” when an issue occurs with your payment gateway or processor – They will Try Again Later, somewhere else.

In retail environments (or in e-commerce environments with large volume) where real-time is required, you need a redundant payment switch that can handle multiple outbound connections to your processors primary and secondary authorization centers (geographical diversity in their data centers) that run on separate connections (routers, outbound copper, and different telcos) but also read this here, Andy talks about this in a little more detail, and talks about geographical diversity that runs though a common CO in an issue that a client had with an authorization provider.

Life before VMWare


I was looking at a few old pictures of my old office at a small Third Party Processor (TPP) where I was Director of Development and Technology and that I lived in for ~8 years, that had a small test lab in my office. These are probably circa 2000-2002 or so, and it required to have physical machine to run different test systems back then. All were white boxes that I put together from parts, let me take a stab at what they did from memory:





Now I have an shuttle PC with 8 gigs of RAM and VMWare ESXi as a test server, and a Vista x64 workstation with 8gb of ram and VMWare Workstation for dev and a Black MacBook for travel with 4gb and VMWare Fusion. I’ve consolidated a bit :) Oh, I also have a XEN instance of Windows 2003 on another Linux box :) BTW, I was an early user of VMware version 3.0 and used it to run Linux guests on my laptops, but that was when RAM was not so cheap, and these boxes had memory limits.

Virtualization rocks for development and testing.

Your browser is out-of-date!

Update your browser to view this website correctly. Update my browser now