Using Python for file processing performance

When doing file processing shell scripting is nice and simple. However when joining two files larger than 2MB, shell scripting performance is poor due to file system IO.

Consider using Python, it uses memory so you can see huge performance improvement.

Example:

#!/usr/bin/env python
# Usage: run.py [PG FILE] [OIM FILE]

import sys, time, smtplib, socket
from datetime import datetime

pg_filename = sys.argv[1]
oim_filename = sys.argv[2]

oim_emp_no_list = []
pg_emp_no_list = []

oim_records = {}

oim_file = open(oim_filename, "rt")
for line in oim_file:
  items = line.split("|")

  if items[0] == "USR_LAST_NAME" : continue
  #if items[73].find("Disabled") == -1 : continue

  oim_record = {}
  oim_record["USR_EMP_NO"] = items[8]
  oim_records[oim_record["USR_EMP_NO"]] = oim_record

  emp_no = oim_record["USR_EMP_NO"]
  if emp_no not in oim_emp_no_list: oim_emp_no_list.append(emp_no)
oim_file.close()

#print "DEBUG> OIM Active OIM count - %s" % len(oim_emp_no_list)

pg_file = open(pg_filename, "rt")
for line in pg_file:
  items = line.split("|")
  pg_record = {}
  pg_record["EmployeeCode"] = items[19]

  emp_no = pg_record["EmployeeCode"]
  if emp_no not in pg_emp_no_list: pg_emp_no_list.append(emp_no)

  try:
    oim_record = oim_records[pg_record["EmployeeCode"]]
  except KeyError:
    print "%s" % pg_record["EmployeeCode"]
    #print "%s" % line
    continue
pg_file.close()

Setting Up Custom XPATH functions in OSB

Custom XPath functions can be setup in OSB11g by following this link : http://eelzinga.wordpress.com/2010/05/10/oracle-service-bus-11g-using-custom-xpath-functions/

Main Steps for deploying custom Xpath functions for Base64 conversion are:

1. Copy the following files - OSBUtilities.jar , OSBUtililies.properties, and OSBUtilities.xml from D:\opt\app\osb11R1\OSB_HOME\config\xpath-functions on soaosbdev1  to the corresponding path in the target environment. If the target environment has multiple machines these files needs to be copied to the $OSB_HOME\config\xpath_functions directory on each of the machine.
2. Update setDomainEnv for each of the domains to include OSBUtilities.jar on Weblogic server's classpath.
3. Restart the whole domain - admin servers and managed servers.

OSBUtilies.jar and the xml and properties file can be extended to include more custom xpath functions in future. Currently the jar contains only 1 class file - Base64Converter.class. New classes implementing new custom functions can be added to  this jar and the xml files and properties file can be extended.

Attaching the source code for Base64 converter which if you want you can put in version control..

Base64Converter.java

public class Base64Converter
{
public static String encodeBase64(String input)
{
try
{
String encoded =new sun.misc.BASE64Encoder().encode(input.getBytes( ));
return(encoded);
}
catch (Exception ex)
{
 return("error");
}
}

public static String decodeBase64(String input)
{
try
{
String decoded =new String(new sun.misc.BASE64Decoder().decodeBuffer(input));
return(decoded);
}
catch (Exception ex)
{
 return("error");
}
}

}

EDIFACT Document Identification with Oracle B2B

When receive a "document protocol identification error" on an inbound native EDI transaction set, it is possible that extraneous characters e.g. CR/LF at the end of the EDI have been added to the file. For example:

The following property setting will have Oracle B2B checks and removes them.

b2b.edi.enablePreprocess= true

The properties will need to be set in the System Mbean Browser within Enterprise manager.
Below are the steps to navigate to the System Mbean Browser, and how to invoke the B2B config Mbean.

1. Login into Enterprise manager.
2. Expand SOA
3. Right click on soa-infra
4. Select Administration
5. Select System Mbean browser
6. Expand Application Defined Mbeans
7. Expand oracle.as.soainfra.config
8. Expand the soa server domain
9. Expand B2B config
10. Select Operations
11. Select addProperty
12. Enter the desired property for the key, such as b2b.edi.enablePreprocess.
13. Enter the value, true.
14. Add any comments if desired.
15. Invoke the mbean and check the b2b-config.xml that the property is reflected in the file.
16. Restart the managed server.

Working with array in Oracle BPEL

This is an extension usage of array based from my previous post.

In the following example the variable Variable_whileLoopIteration is the iteration counter in the WHILE activity.

Reading from a specific element in an array:

<assign name="Extract_ProductName">
<copy>
<from expression="bpws:getVariableData('Variable_PurchaseOrder',concat('/ns1:purchaseOrder/ns1:items/ns1:item[',bpws:getVariableData('Variable_whileLoopIteration'),']/ns1:productName'))"/>
<to variable="Variable_productName"/>
</copy>
</assign>

Writing to a specific element in an array:

<assign name="Assign_NewProductNameToPO">
<copy>
<from expression="'NewProductName'"/>
<to variable="Variable_PurchaseOrder" query="/ns1:purchaseOrder/ns1:items/ns1:item[position() = bpws:getVariableData('Variable_whileLoopIteration')]/ns1:productName"/>
</copy>
</assign>

My advise is don't rely on JDeveloper Design view alone. Most of the time you have to manually edit the source to get it to do exactly what you want.

Convert Gregorian Date to JD Edwards Date in XSLT

Q: How to convert Gregorian Date to JDE Date in XSLT?

The format is C-YY-DDD, which is used by JD Edwards software: the century after 1900, the year in that century and the Julian date within that year.  Here are a few examples:

Checkout JDE Julian Date Converter tool by David Macek.

Important: Make sure you declare the XSLT version 2.0!

<?xml version="1.0" encoding="UTF-8" ?>
<xsl:stylesheet version="2.0">
...
<ns1:JDEDateNumber>
<xsl:variable name="InStringDate" select="'2010-11-09'"/>
<xsl:variable name="Year" select="substring($InStringDate,1,4)"/>
<xsl:variable name="ShortYear" select="substring($InStringDate,3,2)"/>
<xsl:variable name="StartOfYear" select='concat($Year,"-01-01")'/>
<xsl:value-of select="concat('1', $ShortYear, format-number(days-from-duration((xsd:date($InStringDate) - xsd:date($StartOfYear))) + 1, '000'))"/>
</ns1:JDEDateNumber>
...
</xsl:stylesheet>

Configuring Oracle SOA Suite 11g with JDE on AS400 using JDBC

Awhile back I have attempted at using Oracle Application Adapter for JDE without much success, namely because there is no known customer currently. So I'm defaulting to JDBC as a means to connect to our JDE ERP8.

This is a guide for how to configure Oracle SOA Suite 11g to a JD Edwards ERP8 running on AS400 (iSeries). You will need to do these steps to setup your runtime environment.

1. Download/extract IBM Toolkit for Java/JTOpen.
2. Create a new directory, C:\Oracle\Middleware\wlserver_10.3\server\ext\jdbc\as400
3. Copy the jt400.jar to C:\Oracle\Middleware\wlserver_10.3\server\ext\jdbc\as400\jt400.jar
4. Edit C:\Oracle\Middleware\wlserver_10.3\common\bin\commEnv.cmd
5. Append the new jar file in step (3) to the WEBLOGIC_CLASSPATH
   
6. Restart both AdminServer and soa_server1.
7. Goto Oracle Weblogic Administration Console (http://localhost:7001/console/)
8. Navigate to soa_domain > Services > JDBC > Data Sources
9. Add new JDBC Data Source, Name = AS400Connection, JNDI Name = jdbc/AS400Connection, Database Type = DB2 for i5/OS
10. Database Driver as JTOpen's DB2 for i5/OS Driver (Type 4) IBM Toolkit for Java; Version 8.X, 9.X
11. If you choose non-XA, remember to uncheck Supports Global Transactions.
12. When prompted for the database port number, put in a temporary port as 60000.
13. On the Test Database Connection screen, make sure you overwrite the database URL to match your environment, ie. jdbc:as400://JDESERVER;naming=DATASOURCE;errors=full.
14. Deploy to target server soa_server1.
15. If test connection failed with the error message "Cannot load driver: com.ibm.as400.access.AS400JDBCDriver", then repeat step (2) - (6)
16. Next, create a new Outbound Connection Pool Configuration, Interface Name = javax.resource.cci.ConnectionFactory, JNDI Name = eis/DB/AS400Connection, dataSourceName = jdbc/AS400Connection.
17. Save and run Update action on DBAdapter
18. Done!

Configuring JDeveloper 11g with AS400

The following steps is a guide on how to connect to an AS400 database using JDBC and JDeveloper 11g for design-time. Where my local JDeveloper installation is at C:\Oracle\sqldeveloper and my JDE server name is JDEEnt1.

1. Download/extract IBM Toolkit for Java/JTOpen.
2. Copy the jt400.jar to C:\Oracle\sqldeveloper\jdbc\lib\jt400.jar
3. Navigate to Tools > Preferences > Database > JDBC Driver Options
4. Add a new JDBC Driver, calling the driver class as "com.ibm.as400.access.AS400JDBCDriver"
5. For the Library definition, click Browse
6. Name the Library as "AS400 JDBC"
7. Add entry to the class path, referencing the jar file in step (2)
8. Your new AS400 JDBC Driver should look like this
   
9. Back to your SOA composite, simple use the Database Adapter
10. The Database Adapter wizard will prompt you to create a new connection.
    
11. Select Connection Type as "Generic JDBC" and Driver Class as "com.ibm.as400.access.AS400JDBCDriver"
12. Now test the connection
    
    
13. Done!