lunes, 25 de agosto de 2014

IBM Integracion Bus (IIB) COMMIT, ROLLBACK and UNCOORDINATED tests

There aren't many references to the COMMIT, ROLLBACK and UNCOORDINATED keywords in the IBM Integration Bus (IIB; prevously known as WebSphere Message Broker) 9 documentation.

The main reference I have found is in the "The transactional model" section of the official documentation; in the current document version, in the "Database auxiliary transactions" sub-section, it states: "Use the ESQL COMMIT and ROLLBACK statements to commit and roll back auxiliary database transactions. Obtain operations outside the main transaction by specifying the UNCOORDINATED keyword on the individual database statements (for example, the INSERT and UPDATE statements)"; in the "Queue auxiliary transactions"  sub-section, it notes: "The COMMIT and ROLLBACK statements therefore operate only on databases.".

Although the keywords are listed in that sections, I haven't found more specific references (including syntax trees); doing trial and error tests, it can be seen that otherwise as indicated in the documentation, the  UNCOORDINATED keyword can't be specified on the individual database statements:




In the "Dynamic Database Name and Multiple Database Support" IBM presentation (for WebSphere Message Broker v6), the following information can be found: "Any uncommitted database operations that have not been committed by nodes with their transaction property set to COMMIT are automatically committed by the input node if the flow ends normally, and rolled back if there is an unhandled exception".

In an (IBM internal?) presentation (source 1, source 2) syntax trees and usage examples can be found; however, for example, it includes examples of UNCOORDINATED usage that you really can't use in the tool (see imagen above).

According to all this, to my understanding, it can be said you can use the COMMIT and ROLLBACK keywords in some way, to control UNCOORINDATED database transactions (you can mark a transaction as UNCOORDINATED in some way).

After doing several tests, I found the following:
  • The COMMIT and ROLLBACK keywords can be used alone or o accompanied by a specific database  reference (for example, just "COMMIT;" or "COMMIT Database.DSN1;").
  • The UNCOORDINATED keyword can't be used anywhere.
  • In a Compute node accessing databases, the negotations with the database referred in the "Data Source" node property make part of the flow's global (coordinated) transactionality; this transactionality is affected by alone COMMIT and ROLLBACK sentences.
    • You can reference the database referred in the "Data Source" node property in two ways (implicitly or explicitly); the transactionality behaviour for this Database is always the same, no matter which form you use:
      • Implicitly: for example "UPDATE Database.TableX..." or "COMMIT;".
      • Explicitly: for example "UPDATE Database.DSN1.TableX..". or "COMMIT Database.DSN1;", where DSN1 is the value of the "Data Source" property.
  • In a Compute node accessing databases, the negotations with databases other than the referred in the "Data Source" node property (for example INSERT INTO Database.DSNX...) behave as "auxiliary database transactions"; they are affected only by COMMIT or ROLLBACK statements accompanied by the specific database  reference (for example "COMMIT Database.DSNX;").

Test scenary:


Node MQ Input is transactional; it's queue is persistent (with a backout value greater than 0 and a backout queue).


The Compute node contains the following:


The first two sentence groups (X) make reference to the same DSN defined in the node's "Data Source" property (the first group does it implicitly and the second group does it explicitly).

The value of the "Transaction" Compute node property varies between the tests.

The "Compute Error" Compute node executes the following or nothing, depending on the test.


Tests:

Test1: Compute node "Transaction" is "Automatic", COMMIT or ROLLBACK not invoked, flow fails (THROW USER EXCEPTION in "Compute Error" node). 

Result: no data is inserted in any of the databases.

Explanation: no matter the database nor the coordinated or uncoordinated nature, having no COMMIT sentences executed, due to the "Automatic" node transaction type, the flow, after failing, ROLLBACKs to all the pending transactions.




Test2: Compute node "Transaction" is "Automatic", COMMIT or ROLLBACK not invoked, flow doesn't fail (no THROW USER EXCEPTION in "Compute Error" node). 


Result: data is inserted in all databases.

Explanation: no matter the database nor the coordinated or uncoordinated nature, having no COMMIT sentences executed, due to the "Automatic" node transaction type, the flow, after successful finishing, COMMITs to all the pending transactions.




Test3: Compute node "Transaction" is "Commit", COMMIT or ROLLBACK not invoked, flow fails (THROW USER EXCEPTION in "Compute Error" node). 

Expected Result: data is inserted in Table X (TABLAX; Compute node's "Data Source"'s DSN) inmediatly after finishing Compute node execution (it doesn't matter if the entire flow finish with success o failure). No data is inserted in Table Y (TABLAY).

Explanation: the interaction with the main database (X; Compute node's "Data Source"'s DSN) are governed with the "normal semantics", that is, according the success or failure of the node execution (node, not flow, because of the "Commit" value of "Transaction"). The interactions with other databases are governed according this:

Fig X1.

Result: just when the Compute node finish its execution (even when the flow haven't finished), data is inserted in Table X (TABLAX):


In that moment, as expected, Table Y (TABLAY) is empty:


When the flow ends, Table Y (TABLAY) is still empty:



Test4: Compute node "Transaction" is "Commit", COMMIT or ROLLBACK not invoked, flow doesn't fail (no THROW USER EXCEPTION in "Compute Error" node). 



Expected Result: data is inserted in Table X (TABLAX; Compute node's "Data Source"'s DSN) inmediatly after finishing Compute node execution (it doesn't matter if the entire flow finish with success o failure); in that moment, no data is visible in Table Y (TABLAY). When the flow ends, data is inserted in Table Y (TABLAY).

Explanation: the same of test 3.

Result: just when the Compute node finish its execution (even when the flow haven't finished), data is inserted in Table X (TABLAX) (view test 3 image).


In that moment, as expected, Table Y (TABLAY) is empty:


When the flow ends, Table Y (TABLAY) has data:



Test5: Compute node "Transaction" is "Automatic", COMMIT invoked (alone, without DSN specification), flow fails (THROW USER EXCEPTION in "Compute Error" node). 


Expected Result: data is inserted in Table X (TABLAX; Compute node's "Data Source"'s DSN) inmediatly after COMMIT is executed (it doesn't matter if the node or the entire flow finish with success o failure). The COMMIT execution only affects DSN X. No data is inserted in Table Y (TABLAY).

Explanation: the interaction with the main database (X; Compute node's "Data Source"'s DSN) are governed with the "normal semantics", that is, according the success or failure of the node execution or can be affected with COMMIT or ROLLBACK sentences without DSN specification. The interactions with other databases are governed according the stated in FigX1.

Result: just when COMMIT is executed (even when the node haven't finished), data is inserted in Table X (TABLAX).




When the flow ends, the two tables keep with the same state (TABLAX with data, TABLAY empty).

Test6: Compute node "Transaction" is "Commit", COMMIT invoked (alone, without DSN specification), flow fails (THROW USER EXCEPTION in "Compute Error" node). 


The results and explanation are the same of Test 5.

Test7: Compute node "Transaction" is "Commit" or "Automatic", COMMIT invoked (alone, without DSN specification), flow doesn't fail (no THROW USER EXCEPTION in "Compute Error" node). 


Expected Result: data is inserted in Table X (TABLAX; Compute node's "Data Source"'s DSN) inmediatly after COMMIT is executed (it doesn't matter if the node or the entire flow finish with success o failure). The COMMIT execution only affects DSN X. When the flow ends, data is inserted in Table Y (TABLAY).

Explanation: the same as Test5.

Result: just when COMMIT is executed (even when the node haven't finished), data is inserted in Table X (TABLAX).




In that moment, as expected, Table Y (TABLAY) is empty:


When the flow ends, Table Y (TABLAY) has data:



Test8: Compute node "Transaction" is "Automatic", COMMIT invoked (with DSN specification for database Y), flow doesn't fails (no THROW USER EXCEPTION in "Compute Error" node). 


Expected Result: data is inserted in Table Y (TABLAY) inmediatly after COMMIT is executed (even if the node hasn't finished execution; it doesn't matter if the node or the entire flow finish with success o failure). The COMMIT execution only affects DSN Y. No data is inserted in Table X (TABLAX) because the flow fails.

Explanation: the interaction with the main database (X; Compute node's "Data Source"'s DSN) are governed with the "normal semantics", that is, according the success or failure of the node or flow execution or can be affected with COMMIT or ROLLBACK sentences without DSN specification; they are obviously not affected by COMMITs accompanied by a specific DSN reference. The interactions with other databases are governed according the stated in FigX1.

Result: just when COMMIT is executed (even when the node haven't finished), data is inserted in Table Y (TABLAY).




In that instant, TABLAX is empty.


When the flow ends, TABLAX is still empty.



Test9: Compute node "Transaction" is "Commit", COMMIT invoked (with DSN specification for database Y), flow fails (THROW USER EXCEPTION in "Compute Error" node). 


Expected Result: data is inserted in Table Y (TABLAY) inmediatly after COMMIT is executed (even if the node hasn't finished execution; it doesn't matter if the node or the entire flow finish with success o failure). The COMMIT execution only affects DSN Y. Once the node finishes its execution, data is inserted im Table X (TABLAX); it doesn't matter if the entire flow finish with success o failure.

Explanation: the same as Test8.

Result: just when COMMIT is executed (even when the node haven't finished), data is inserted in Table Y (TABLAY).





In that instant, TABLAX is empty.


When the node ends, TABLAX has data

.



Test10: Compute node "Transaction" is "Commit" or "Automatic", COMMIT invoked (with and without DSN specification for database Y), flow fails (THROW USER EXCEPTION in "Compute Error" node). 


Expected Result: data is inserted in Table Y (TABLAY) and Table X (TABLAX) inmediatly after COMMITs are executed (even if the node hasn't finished execution; it doesn't matter if the node or the entire flow finish with success o failure).

Explanation: the same as Test8.

Result: just when COMMITs are executed (even when the node haven't finished), data is inserted in Table X (TABLAX) and Table Y (TABLAY).











Test11: Compute node "Transaction" is "Commit" or "Automatic", ROLLBACK invoked (without DSN specification), flow doesn't fail (no THROW USER EXCEPTION in "Compute Error" node). 



Expected Result: data is inserted in Table Y (TABLAY) after the flow ends. Table X (TABLAX) is empty.

Explanation: the same as Test8.

Result: 



Regardless the node transactionality configuration, the data in Table Y (TABLAY) is only commited when the flow finalizes (not the node).


Test12: Compute node "Transaction" is "Commit" or "Automatic", ROLLBACK invoked (with and without DSN specification), flow doesn't fail (no THROW USER EXCEPTION in "Compute Error" node). 


Expected Result: both tables are empty when the flow fhinishes.

Explanation: the same as Test8.

Result: 




Test13: Compute node "Transaction" is "Automatic", ROLLBACK and COMMIT invoked (with and without DSN specification), flow doesn't fail (no THROW USER EXCEPTION in "Compute Error" node). 


Result: 




Test14: COMMIT and ROLLBACK placed in not permitted places (Compute node without Data Source). 

The following error is generated (in spanish):

'[Microsoft][Administrador de controladores ODBC] No se encuentra el nombre del origen de datos y no se especificó ningún controlador predeterminado'




lunes, 30 de junio de 2014

IBM Integration Bus (Message Broker) resources

Steps to configure an Oracle ODBC data source on Windows: "Connecting to a database from Windows systems". Includes necessary steps like "Select Enable SQLDescribeParam", "Select Procedure Returns Results", "Select Login Timeout and set the value to 0" and the WorkArounds registry string with 536870912 value, which you may forget, causing you unnecesary pain and suffering. Personally, I was struggling with the '[IBM][ODBC Oracle Wire Protocol driver]Data type for parameter x has changed since first SQLExecute call.' error (which always ocurred the second time I executed an specific flow, and that was caused by the lack of the registry key) and the very generic "[Microsoft][ODBC Driver Manager] Driver does not support this function" error (forum references: first SQLExecute call Error and ODBC Oracle Wire Protocol driver Error).

SQLSTATE explanation: ESQL: What is the SQLSTATE?

Explanation about the difference between a REFERENCE to OutputRoot and the real OutputRoot: "Message Headers not getting copied to output" 3rd post (useful when you use "IN REFERENCE OutputRoot" in functions or procedures)

Explanation of the logic behind "All calls to CURRENT_TIMESTAMP within the processing of one node are guaranteed to return the same value":  "a node is an interruption to the execution path of a message. At the time the node is called, the thread is suspended and a call to get current timestamp retrieves the time value stored in the environment at the time the execution path was interrupted. If you want to measure time between certain points within your ESQL logic of the same node, you can do so by calling Date.getTime() Java api call from ESQL" (source: Very strange issue with CURRENT_TIMESTAMP, lancelotlinc user post).

List of Websphere Message Broker resources that changed its name in IBM Integration Bus: Do you want to know the new name of a WebSphere Message Broker resource that has changed its name in IBM Integration Bus v9?

Implicit casting rules from database data types to ESQL data types: Data types of values from external databases

Standard output (stdout) (Windows 7): C:\ProgramData\IBM\MQSI\components\[BROKER_NAME]\[ID]\console.txt

ESQL-to-Java data-type mapping table.

Log4j troubleshooting: keep an eye on the stdout file (console.txt, see above); when you call the initLogger method (com.ibm.broker.IAM3.Log4jNode.initLog4j) you should see a line like the following in that file:

2015-03-05 21:56:31.384     17 log4j:WARN Config file used: /C:/Lib/iib-log4j/conf/brokerlog.xml

You should also find there information about problems finding or parsing that file. If you have checked everything and still can't see logs entries in the log file, try using a tool like Process Explorer (for Windows; it shows you all file access events for all running process) and filter for events which "Path" contains some pattern your log file name has or just filter for events for "DataFlowEngine.exe" process name; look for example what I found in my case:


I was not aware that I was specifying the "File" param value for my file appender using backslashes (C:\Logs\xxx.log) which resulted in this file for IIB: C:\Program Files\IBM\MQSI\9.0.0.1\bin\Logsxxx.log. Once I changed the "File" param value for the windows path using slashes (C:/Logs/xxx.log) I could see events in my log file.

Dynamic Database Name and Multiple Database Support: IBM presentation, Accessing databases from ESQL, "ESQL Enhancements Part 1" presentation (other link). The last link contains the syntax trees for UNCOORDINATED database references and DSN specific COMMITs and ROLLBACKs; I haven't found references and/or examples of that syntax in other places; the "The transactional model" section of the official documentation says you can use UNCOORDINATED, COMMITs, ROLLBACKs in "Database auxiliary transactions", but I don't see this reflected in the syntax threes in that same documentation (IBM, why are you like this? :( ).

How to change the WebSphere MQ Explorer (from IIB 9)  language to english: add -Duser.language=en to C:\Program Files (x86)\IBM\WebSphere MQ Explorer\MQExplorer.ini (source).

ODBC SQLSTATE codes:
Example procedure to navigate a message tree (including the Exception List): Navigating a message tree (including the Exception List).

IBM Integration Bus (IIB) MQ retry and requeue tests.

IBM Integracion Bus (IIB) COMMIT, ROLLBACK and UNCOORDINATED tests.

DeverloperWorks' Top 12 IBM Integration Bus articles.


lunes, 9 de junio de 2014

Cambiar el idioma de IBM Integration Toolkit 9 a inglés

Por defecto el idioma de la UI de IBM Integration Toolkit 9 (de IBM Integration Bus) UI está determinado por la local actual. Mi OS está instalado en Español, pero prefiero usar las herramientas de desarrollo en inglés (mensajes de error, tutoriales, etc.)

Para forzar IBM Integration Toolkit 9 a usar inglés, añadir la siguiente línea

-nl en_US

al archivo INSTALLATION_PATH\mb.ini (por ejemplo C:\Program Files\IBM\IntegrationToolkit90\mb.ini)