uCosminexus Service Platform, Overview

[Contents][Glossary][Index][Back][Next]

8.7.3 File conversion operation

This subsection describes the functionality and file formats supported in the file conversion operation.

Organization of this subsection
(1) Overview of functionality of the file conversion operation
(2) Supported I/O file formats
(3) Batch processing method
(4) Partition processing method
(5) How to specify the file formats to be converted
(6) How to specify the data transformation definition
(7) File conversion using character code conversion UOC
(8) Specifying a self-defined file in the partition processing method

(1) Overview of functionality of the file conversion operation

The file conversion operation reads the conversion source input file from the work folder or common folder, and writes the post conversion output file to the work folder or common folder.

The following figure shows an overview of the processing of the file conversion operation:

Figure 8-26 Processing of the file conversion operation

[Figure]

(2) Supported I/O file formats

The following table describes the formats of the input and output files that are supported in the file conversion operation:

Table 8-12 File formats supported in the file conversion operation

No. Processing method Input file format Output file format Support
1 Batch processing method Binary format Binary format Y
2 XML format Y
3 XML format Binary format Y
4 XML format Y
5 Partition processing method Binary format Fixed length format Binary format Y
6 XML format Y
7 Length tag format Binary format N
8 XML format N
9 linefeed separator format Binary format Y
10 XML format Y
11 Other than above Binary format N
12 XML format N
13 XML format Binary format Y
14 XML format Y

Legend:
Y: Supported.
N: Not supported.

The differences between the fixed length, length tag, and linefeed separator formats are as follows:

You can convert the layout when you convert the file. Also, if the I/O format is the binary format, you can also assign or delete a linefeed code and convert the character code.

Reference note
If the input file contains even 1-byte data, file conversion is executed (even if the data only contains null, linefeed character, or EOF character, file conversion is executed).

(3) Batch processing method

In the batch processing method, the input files are read and converted in a batch. After conversion, the data is written to the output file in a batch.

The target data format, and the contents to be converted in the batch processing method are as follows:

(4) Partition processing method

The reading of the input file, conversion, writing to the output file is processed for each record of the input file. Repetitive processing is executed until the end of the input file.

For examples of definition when a file is converted from the binary format to XML format, and from the XML format to binary format with the partition processing method, see "Appendix E Example for defining file operations adapter " in "Service Platform Reception and Adapter Definition Guide".

(a) Types of judgment methods for each record

The judgment method for each record varies depending on the input file format.

In the binary format
The judgment for each record includes the following methods:
  • Fixed length method
    This method specifies the byte length of the record in advance. Use this method when the input file is in the fixed length format.
  • Linefeed separator method
    This method considers the occurrence of the linefeed code (LF (0x0a) or CR+LF (0x0d 0x0a)) as a record separator. Use this method when the input file is in the linefeed separator format. For details on the linefeed separator method, see the description in "(d) Partitioning the records (when the input file is in the binary format)".
    If the linefeed separator method is selected in the partition processing method, regardless of the binary format definition, the separation of records is determined on the occurrence of the linefeed code (LF (0x0a) or CR+LF (0x0d 0x0a)), so when the linefeed byte pattern of the character code existing in the conversion source file is not a linefeed code (examples of character code: UTF-16_BIG, UTF-16_LITTLE, KEIS+EBCDIC, KEIS+EBCDIK, and KEIS), you cannot perform normal processing in the partition processing method of the linefeed separator method.
    Note that if you select CUSTOM, the processing depends on the data to be processed.

In the XML format
Specify the elements to be handled as records in the file operations adapter definition file.
Note
The sizes of the input file and output file are not limited. However, a file, where the size of one record in the input file and output file is 500 MB or more, cannot be converted regardless of the input file format. The operations are not guaranteed if you convert a file where the size of one record is 500 MB or more. Also, if the size of one element configuring a record exceeds 500 MB or more, the operation is not guaranteed regardless of the input file format.
(b) Target data formats

The data formats of the input file subject to the partition processing method are the binary format definition and XML schema. The target data structure must be a structure in which the header record part, data record part, and trailer record part occur in this sequence.

The data record part is a data having a structure wherein the decided element sequence is continuously repeated. This structure is the same as the database table structure, the decided element sequence (lines in the binary format, and data beneath specific elements in the XML format) is called a record, and this is an image that is repeated n times. There are no restrictions on the number of elements in a record and the number of records.

The header record part and trailer record part are complex content elements containing any element, and the records are not repeated. Furthermore, you can omit these parts.

The following figures show the images of the data file structure and data record part:

Figure 8-27 Data file structure

[Figure]

Figure 8-28 Image of the data record part

[Figure]

The data format corresponds to the binary format (fixed length format and linefeed separator format), and XML format. The input and output formats must have the structure of the above formats.

The following figures show the examples of data in the binary format:

Figure 8-29 Example of data in the fixed length format

[Figure]

Figure 8-30 Example of data in the linefeed separator format

[Figure]

(c) Files required for the partition processing method

The binary format definition, XML schema, and data transformation definition that must be defined in the partition processing method are as follows:

Binary format definition and XML schema
The required definitions vary depending on the format of the input file and output file.
If the input file or output file is in the binary format
The file formats on the input side and output side must be defined, so use the binary format definition editor to create the required formats from among the following formats:
  • Format of the header record part on the input side
  • Format of the data record part on the input side
  • Format of the trailer record part on the input side
  • Format of the header record part on the output side
  • Format of the data record part on the output side
  • Format of the trailer record part on the output side
If the input file or output file is in the XML format
Create the XML schema for the input side and output side. Unlike the binary format, you can define the header record, data record, and trailer record in one XML schema. The XML schema is required for creating the data transformation definition, but is not used for executing the transformation operation.

Data transformation definition
You must define the data transformation between the formats on the input side and output side, so use the mapping definition editor to create the following data transformation definitions:
  • Data transformation definition for the header record part
  • Data transformation definition for the data record part
  • Data transformation definition for the trailer record part
Note that if the output side is the XML format, you can perform mapping by limiting the items beneath the complex content element that specifies the mapping target. If the header record, data record, and trailer record are defined in one XML schema, you can generate the data transformation definition for each record part by limiting the items beneath the complex content element that forms the root element of each record and performing mapping. For details, see "6.4.8 Restricting mapping range " in "Service Platform Basic Development Guide".
(d) Partitioning the records (when the input file is in the binary format)

Judgment method for records
You can define record demarcation by selecting from fixed length format or linefeed separator format.
Specify the type of the fixed length format or linefeed separator format with the following parameter in the file operations adapter definition file:
  • csc.adapter.fileOperation.transform.format
For the contents, see "File operations adapter definition file" in "Service Platform Reference Guide".
  • For the fixed length method
    The record demarcation is the position specified in the following parameters of the file operations adapter definition file:
    [Figure]csc.adapter.fileOperation.transform.header.fixedSize
    [Figure]csc.adapter.fileOperation.transform.data.fixedSize
    [Figure]csc.adapter.fileOperation.transform.trailer.fixedSize
  • For the linefeed separator method
    The record demarcation is the position where the linefeed code (LF (0x0a) or CR+LF (0x0d 0x0a)) occurs in the input file. The end of file is also handled as the record demarcation. Note that if the file end occurs immediately after a linefeed, this is handled as one record separator along with the last linefeed.

Location and length of the record part
In the partition processing method, you must determine which record is currently being read from among the header record part, data record part, and trailer record part. The methods for determining the position and length of the record part for the fixed length format and linefeed separator format are as follows:
  • In the fixed length method
    The following table describes the determination for the position and length of the record part:

    Table 8-13 Record types depending on the record presence specification (Fixed length method)

    Record presence specification Presence of header record part Present Present Absent Absent
    Presence of trailer record part Present Absent Present Absent
    Position and length of the record part Header record part Top of file
    to
    Header record length
    Top of file
    to
    Header record length
    -- --
    Data record part Header record part
    End position
    to
    Data record length
    * Number of data records
    Header record part
    End position
    to
    Data record length
    * Number of data records
    Top of file
    to
    Data record length
    * Number of data records
    Top of file
    to
    Data record length
    * Number of data records
    Trailer record part Data record part
    End position
    to
    Trailer record length
    -- Data record part
    End position
    to
    Trailer record length
    --

    Legend:
    --: Not applicable.

    Obtain the number of data records with the following formula:
    Data record part size =
    Data file size - (Header record length + Trailer record length)
    Number of records = Size of data record part / Data record length (rounded off after the decimal point)
    If there is a remainder in the calculation results of the number of records, a message (KDEC80045-W) is output and then the processing continues. In this case, the possibilities are as follows, so check and eliminate the cause:
    EOD exists at the end of the input file.
    There is an error in the input file.
    There is an error in the settings for the following items in the file operations adapter definition file:
    csc.adapter.fileOperation.transform.headerRecord
    csc.adapter.fileOperation.transform.trailerRecord
    csc.adapter.fileOperation.transform.header.fixedSize
    csc.adapter.fileOperation.transform.data.fixedSize
    csc.adapter.fileOperation.transform.trailer.fixedSize
  • In the linefeed separator method
    The following table describes the determination for the position and length of the record part:

    Table 8-14 Record types depending on the record presence specification (Linefeed separator method)

    Record presence specification Presence of header record part Present Present Absent Absent
    Presence of trailer record part Present Absent Present Absent
    Position and length of the record part Header record part First line First line -- --
    Data record part First line onwards
    to
    Just before last line
    First line onwards
    to
    Last line
    First line
    to
    Just before last line
    First line
    to
    Last line
    Trailer record part Last line -- Last line --

    Legend:
    --: Not applicable.

    If the message (KDEC80047-E) is output and the processing is terminated, the possibilities are as follows, so check and eliminate the cause:
    The number of records in the input file is not sufficient.
    There is an error in the settings for the following items in the file operations adapter definition file:
    csc.adapter.fileOperation.transform.headerRecord
    csc.adapter.fileOperation.transform.trailerRecord

    Table 8-15 Error determining pattern

    Header record part Trailer record part Number of lines to be converted in the file
    Yes Yes 1 line or less
    Yes No 0 lines or less
    No Yes 0 lines or less
(e) Partition of records (When the input file is in the XML format)

To specify the record demarcation of the header record, data record, and trailer record for the input XML file, set up the path of the elements to be read in the file operations adapter definition file.

The following table describes the items and contents set up in the file operations adapter definition file:

Table 8-16 Settings in the file operations adapter definition file (Partition input in the input XML file)

Set items Set contents
csc.adapter.fileOperation.transform.header.inElement Specify the path of the elements corresponding to the header record in the input XML file.
If a header record is not used, this item need not be specified (ignored even if specified).
csc.adapter.fileOperation.transform.data.inElement Specify the path of the elements corresponding to the data record in the input XML file.
csc.adapter.fileOperation.transform.trailer.inElement Specify the path of the elements corresponding to the trailer record in the input XML file.
If a trailer record is not used, this item need not be specified (ignored even if specified).

The following figure shows the example settings of the input XML file and file operations adapter definition file:

Figure 8-31 Example settings of the input XML file and file operations adapter definition file

[Figure]

(f) Merging of records (When the output file is in the XML format)

In the partition processing method, a output template XML file is used for the input file in order to merge the conversion results of each record into one XML file.

The contents of the output template XML file must be formatted according to the output XML schema created in "(b) Target data formats". If the file is not an integral-form XML document, an error occurs during conversion.

The following table describes the specifications for the output template XML file:

Table 8-17 Specifications for the output template XML file

Item Specifications
File name File name specified in the csc.adapter.fileOperation.transform.outputTemplateXmlFile property of the file operations adapter definition file
Character encoding UTF-8

Specific elements in the output template XML file are replaced in the records (output data), to create the output XML file.

Specify the elements replaced in the records in the file operations adapter definition file. The following table describes the items and contents set up in the file operations adapter definition file:

Table 8-18 Settings in the file operations adapter definition file (Merging the output XML file)

Set items Set contents
csc.adapter.fileOperation.transform.header.outElement Specify the path of the elements corresponding to the header record in the output XML file.
If a header record is not used, this item need not be specified (ignored even if specified).
csc.adapter.fileOperation.transform.data.outElement Specify the path of the elements corresponding to the data record in the output XML file.
csc.adapter.fileOperation.transform.trailer.outElement Specify the path of the elements corresponding to the trailer record in the output XML file.
If a trailer record is not used, this item need not be specified (ignored even if specified).

The following figure shows the example settings of the output template XML file, and file operations adapter definition file:

Figure 8-32 Example settings of the output template XML file and file operations adapter definition file

[Figure]

The following figure shows an example of converting the output template XML file into the output XML file. Note that the linefeed and spaces of the output XML file are displayed in this example, but in the actual output XML file, the linefeed and spaces between the elements are deleted.

Figure 8-33 Example of XML file output

[Figure]

The output XML file that uses the output template XML file is output in the format described in the following table:

Table 8-19 Format of the output XML file

Item Specifications
Character encoding UTF-8
Spaces between the elements Deleted.
Linefeeds between the elements Deleted.
Namespace declaration
  • The Namespace declaration of the output template XML file is inherited.
  • Un-used Namespace declaration is not deleted.
  • The missing Namespace declaration is supplemented with input XML declaration.
File contents XML data is output according to the output XML schema. The compliance with the output XML schema is not validated.
Note
  • If you specify character encoding that is not supported by XML Processor, an error occurs during parsing. For the character encoding supported by XML Processor, see "1.3.2 Character codes that can be processed" in "Application Server XML Processor User Guide".
  • If elements not corresponding to any header, data, and trailer records are defined in the output template XML file, the elements are output as along with the value. Even if there are spaces before and after the value, the spaces are not removed.
  • In the data record, the element occurring first, and the sibling elements occurring in continuation are handled as one data record group. After the data record group is terminated, there is an error if a data record occurs again.
  • When the contents of the output template XML file are not formatted according to the output XML schema created in "(b) Target data formats", even if you specify the correct value existing in the output template XML file for the element replaced in the record, the record conversion results are not replaced and the processing is terminated. In this case, a runtime error does not occur. Therefore, even if the settings specify that the input data be output with the data transformation definition, if the record conversion results are not output to the output file, review and, if necessary revise, the contents of the output XML schema and output template XML file.
(g) Simultaneous conversion of multiple data records

If the partition processing method is selected in the file conversion operation, you can collectively convert multiple data records as one element. You can perform the conversion whether the input file is in the binary format or XML format.

The following figures show examples where the input file is in the binary format, the total number of data records is 6, and two data records are converted as one element:

Figure 8-34 Example of converting two data records as one element (in the fixed length format)

[Figure]

Figure 8-35 Example of converting two data records as one element (in the linefeed separator format)

[Figure]

Reference note
In the linefeed separator format, the conversion includes the linefeed of the data records.

Specify the settings for one element (number of records converted simultaneously) in the following parameter of the file operations adapter definition file:

For the contents of the parameter, see "File operations adapter definition file" in "Service Platform Reference Guide".

Note
If you convert multiple data records simultaneously, you improve the conversion efficiency, but if you enlarge one element, the processing performance might decline. Using 500 MB as a guide for the upper limit of data size per element, appropriately specify the number of records to be converted simultaneously.

If the total number of data records is indivisible by the value specified in the parameter, convert the remaining records at one time.

The following figure shows an example of conversion where the total number of data records is 6, and four data records are converted as one element:

Figure 8-36 Example of conversion when the records are indivisible by the value specified for the element

[Figure]

(h) Simultaneously converting multiple header records

When all the following conditions are fulfilled, you can collectively convert multiple header records as one element:

You can specify the number of header records to be converted in the csc.adapter.fileOperation.transform.header.batchSize property of the file operations adapter definition file.

In the following conditions, an error might occur during conversion:

(5) How to specify the file formats to be converted

Specify the format of the files to be read or written in the file conversion operation by first creating a binary format definition file on the binary format definition screen, and then specifying the file format as a self-defined file of the file operations adapter.

For details on the binary format definition screen, see "1.2.1 Binary Format Definition Window" in "Service Platform Reference Guide".

For details on creating the binary format definition file, see "4.4 Creating Message Formats (Binary Format Definition File)" in "Service Platform Basic Development Guide".

For details on specifying the self-defined file of the file operations adapter, see "File operations adapter definition file" in "Service Platform Reference Guide".

(6) How to specify the data transformation definition

Specify the data transformation definition file (xsl file) to be converted from the pre-created input file to the output file, as the self-defined file of the file operations adapter.

To create a mapping definition file with the mapping definition editor for the development environment, use the data transformation definition to create the xsl file. For details on specifying the self-defined file of the file operations adapter, see "File operations adapter definition file" in "Service Platform Reference Guide".

(7) File conversion using character code conversion UOC

In the file conversion operation, you can use the character code conversion UOC to convert files.

When you use the character code conversion UOC, you must set up the following files for the self-defined file on the Service Adapter Settings window (details):

  1. To use the same self-defined file for the conversion of the input file and output file
    • csc_owncodeconvert.properties
  2. To use different self-defined files for the conversion of the input file and output file
    • csc_owncodeconvert_in.properties
    • csc_owncodeconvert_out.properties

If you specify the file described in 1 at the same time as the file described in 2, the file described in 1 is given priority and the file described in 2 is ignored.

For details on how to specify the settings in the Service Adapter Settings window (details), see the locations describing the file conversion operation in "3.3.11 Defining file operations adapters" in "Service Platform Reception and Adapter Definition Guide".

For an overview of the character code conversion UOC and the implementation method, see "Appendix I Character code conversion using character code conversion UOC" in "Service Platform Basic Development Guide".

(8) Specifying a self-defined file in the partition processing method

The following table describes the self-defined files required for converting the partition processing method in the file conversion operation, and the list of properties of the file operations adapter definition file that must be specified. For details on the file operations adapter definition file, see "File operations adapter definition file" in "Service Platform Reference Guide".

Table 8-20 Self-defined files and properties of file operations adapter definition file that must be specified in the partition processing method

Files specified as self-defined file Input file Output file
Binary XML Binary XML
XSLT file for header layout conversion Y#1 Y#1 Y#1 Y#1
XSLT file for data layout conversion Y Y Y Y
XSLT file for trailer layout conversion Y#2 Y#2 Y#2 Y#2
FDX file for input header Y#1 -- -- --
FDX file for input data Y -- -- --
FDX file for input trailer Y#2 -- -- --
XML schema file for input header -- -- -- --
XML schema file for input data -- -- -- --
XML schema file for input trailer -- -- -- --
Properties specified in the file operations adapter definition file csc.adapter.fileOperation.transform.input
(input file type)
Y Y -- --
csc.adapter.fileOperation.transform.output
(output file type)
-- -- Y Y
csc.adapter.fileOperation.transform.headerRecord
(use of header records)
Y#1 Y#1 Y#1 Y#1
csc.adapter.fileOperation.transform.trailerRecord
(use of trailer records)
Y#2 Y#2 Y#2 Y#2
csc.adapter.fileOperation.transform.header.inElement
(name of input header element)
-- Y#1 -- --
csc.adapter.fileOperation.transform.data.inElement
(name of input data element)
-- Y -- --
csc.adapter.fileOperation.transform.trailer.inElement
(name of input trailer element)
-- Y#2 -- --
csc.adapter.fileOperation.transform.header.outElement
(name of output header element)
-- -- -- Y#1
csc.adapter.fileOperation.transform.data.outElement
(name of output data element)
-- -- -- Y
csc.adapter.fileOperation.transform.trailer.outElement
(name of output trailer element)
-- -- -- Y#2
csc.adapter.fileOperation.transform.outputTemplateXmlFile
(file name of output template XML file)
-- -- -- Y
FDX file for output header -- -- Y#1 --
FDX file for output data -- -- Y --
FDX file for output trailer -- -- Y#2 --
XML schema file for output header -- -- -- --
XML schema file for output data -- -- -- --
XML schema file for output trailer -- -- -- --
Output template XML file -- -- -- Y
Input property for character code conversion UOC (common I/O file)
csc_owncodeconvert.properties
Y#3 -- Y#3 --
Input property for character code conversion UOC (for converting the input file)
csc_owncodeconvert_in.properties
Y#3 -- -- --
Input property for character code conversion UOC (for converting the output file)
csc_owncodeconvert_out.properties
-- -- Y#3 --

Legend:
Y: Must be specified.
--: Need not be specified.

#1
Specify this file or property only when a header record is used.

#2
Specify this file or property only when a trailer record is used.

#3
Specify this property when the character code conversion UOC is used.