diff --git a/StudentProject-Readme.md b/StudentProject-Readme.md new file mode 100644 index 00000000000..8bef586f5ee --- /dev/null +++ b/StudentProject-Readme.md @@ -0,0 +1,510 @@ +# Final Report + +This report covers three related development tasks. The first task implemented getter and setter functionality for frame column names as part of SYSTEMDS-3857. During its validation, a separate defect in CSV header generation was identified and fixed. Further propagation tests then revealed broader inconsistencies in the runtime +handling of column-name metadata, which motivated the third task and the corresponding redesign. + +--- + +# Getters and Setters + +The following sections describe the work carried out as part of the original Ticket: **SYSTEMDS-3857**, +including the development approach, the implementation of the new built-in functions, and their validation. + +## Objective + +The objective of the initial ticket **SYSTEMDS-3857** was to implement and test getter and setter +functionality for frames in SystemDS. + +Accessing and modifying column names is a common operation in data processing workflows. +Similar functionality is provided by widely used data analysis frameworks such as Pandas and R, +where column names are treated as an integral part of a data frame's metadata. + +## Initial Architecture + + +Prior to this work, SystemDS exposed the built-in function `colnames()` to DML for retrieving +the column names of a frame. However, no complementary operation was available to modify column +names. Consequently, DML provided only read access to this aspect of frame metadata, without a +corresponding write operation. + +In contrast, data analysis environments such as Pandas and R provide a symmetric interface +for both retrieving and assigning column names. Establishing the same symmetry in the SystemDS +DML interface was therefore one of the primary objectives of the initial ticket. + + +## Analysis + +Although the necessary functionality for setting column names already existed +internally, there was no way to access or use these operations directly from DML code. +This ticket addressed that limitation by exposing the corresponding functionality to the +DML language and validating it through appropriate tests. + +### Approach + +The development process followed a top-down, trace-driven approach. As a starting point, +a DML script was created that called the desired function, even though the +functionality had not yet been implemented. Executing this script revealed +missing components in the processing pipeline. + +The execution path was then traced with the help of a debugger and the behavior was compared with that of existing built-in functions implementing +similar functionality, such as `colnames(x)`. Whenever a missing implementation or unsupported +code path was encountered, the corresponding component was analyzed, extended, and integrated +before continuing with the next execution step. + +This iterative process made it possible to identify all components involved in the execution +of the new built-in functions and to implement the required functionality incrementally across +the different layers of the SystemDS architecture. + + +```mermaid +flowchart LR + + A["DML Script"] + B["Builtins"] + C["BuiltinFunctionExpression"] + D["DMLTranslator"] + E["InstructionUtils"] + F["CP Instruction"] + + A -->|"getNames() / setNames()"| B + B --> C + C --> D + D --> E + E --> F + + B1["Register new built-ins"] + C1["Validate function call"] + D1["Create runtime instruction"] + E1["Map opcode to instruction"] + F1["Execute metadata operation"] + + B -.-> B1 + C -.-> C1 + D -.-> D1 + E -.-> E1 + F -.-> F1 + + style B1 fill:,stroke-dasharray:3 3 + style C1 fill:,stroke-dasharray:3 3 + style D1 fill:,stroke-dasharray:3 3 + style E1 fill:,stroke-dasharray:3 3 + style F1 fill:,stroke-dasharray:3 3 +``` + +### Implementation + +The implementation of the `getNames()` and `setNames()` built-in functions required changes across multiple layers of the SystemDS architecture. The following table summarizes the relevant classes, their purpose, and the modifications introduced. + +| Class | Method | Purpose | Changes | +|-------|---------|---------|---------| +| `Builtins` | - | Defines all built-in DML functions. | Registered `getNames()` and `setNames()`. | +| `Opcodes` | - | Defines runtime opcodes for instructions. | Added opcodes for the new built-in functions. | +| `BuiltinFunctionExpression` | - | Parses and validates built-in function calls. | Added parsing support for the new functions. | +| `DMLTranslator` | - | Translates DML expressions into runtime instructions. | Added translation of the new built-ins into CP instructions. | +| `InstructionUtils` | - | Parses instruction strings and creates runtime instructions. | Registered the new instruction types. | +| `UnaryFrameCPInstruction` | `processInstruction()` | Executes unary frame instructions. | Implemented `getNames()`. | +| `BinaryFrameFrameCPInstruction` | `processInstruction()` | Executes binary frame instructions. | Implemented `setNames()`. | + +### Validation +The implementation was verified using round-trip tests. +Column names are first written to a frame using `setNames()` and +subsequently read back using `getNames()`. +These tests were executed in both CP and Spark execution modes to ensure +consistent behavior across different execution environments. +Their purpose is to verify that the assigned column names are +preserved correctly and that no information is lost or modified during the round-trip process. + +### Documentation + +The behaviour and usage of this implementation are documented in the corresponding `dml-language-reference.md` + + +--- + +# CSV-Header Bug Fix + +During the implementation and testing of the column-name functions, a separate defect was +discovered in CSV header generation. +The bug caused incorrect header generation under specific conditions and was +investigated and fixed in a separate branch/PR. +The following sections describe the underlying problem, +its root cause, and the implemented solution. + +### Objective and Initial Architecture +During CSV export, the header row was generated using incorrect array indices. +Since column names are stored using zero-based indexing, iterating from `1` to `<= numColumns` +caused the first column name to be skipped and resulted in an out-of-bounds access for the +last iteration. The objective of this task was to implement and test a corresponding fix. + +#### Original Implementation + +```java +for (int j = 1; j <= blk.getNumColumns(); j++) { + sb.append(blk.getColumnNames()[j] + + ((j < blk.getNumColumns() - 1) ? _props.getDelim() : "")); + } +``` + +### Analysis + +The bug was investigated using a test-guided debugging approach similar to the implementation +of the getter and setter implementation. +First, a dedicated test case was given to reproduce the incorrect CSV header behavior reliably. +The test constructs a `FrameBlock` with predefined column names, writes it to a CSV file +with an enabled header, and reads the generated output back. + +The failing test was then executed with a debugger. +By tracing the CSV write path step by step, the header generation logic +was identified as the source of the error. Comparing the loop bounds with the +zero-based indexing of the `columnNames` array revealed the off-by-one error in `FrameRDDConverterUtils`. + +### Approach and Implementation + +The implementation was corrected by using zero-based iteration (`0` to `< numColumns`) +and adjusting the delimiter condition accordingly. + + +#### Corrected Implementation + +```java +for (int j = 0; j < blk.getNumColumns(); j++) { + sb.append(blk.getColumnNames()[j]) + .append(j < blk.getNumColumns() - 1 ? _props.getDelim() : ""); + } +``` + +### Validation + + +The fix for this bug was validated by a regression test. +The test verifies that explicitly assigned frame column names are preserved throughout a +complete CSV processing pipeline. + +First, a FrameBlock with three columns is created using the +schema FP64. The frame is assigned the custom column names customer_id, +signup_date, and score. It is then populated with 42 rows of randomly generated data. + +The input frame is subsequently written to a CSV file with header generation enabled. +The corresponding DML script reads the generated CSV input and writes the resulting +frame back to another CSV file. The output file is then read again into a FrameBlock, +and the resulting column names are compared with the original array using +`Assert.assertArrayEquals`. + +With the previous implementation, the test failed because the generated +CSV header did not contain the complete set of column names. +The first column name was omitted, and the final iteration attempted to access an +index beyond the array bounds. After applying the bug fix, the test passes in both +execution modes and confirms that all column names are preserved correctly. +Since the test is part of the automated test suite, it also serves as a regression test. +Future modifications to the CSV writer or frame I/O implementation will therefore be checked +against the expected behavior, greatly reducing the risk that the same off-by-one error is introduced +again in the future. + +--- + +# Metadata Handling +The implementation of getter and setter functions exposed several inconsistencies in the handling of frame column-name metadata.. +During testing it became apparent that column names were not propagated reliably across all investigated frame operations. +This motivated a more comprehensive analysis of the metadata architecture presented in this chapter. + +This chapter analyzes how frame metadata is represented and managed in the original +SystemDS implementation. Understanding the existing architecture is essential for +identifying the limitations that motivated the redesign of the column names later in this chapter. + +## Problem Description + +The original implementation stores column names inside each individual FrameBlock rather than at the frame level, +which introduced several challenges. +Whenever new frame blocks are created during execution, the associated metadata must be +propagated explicitly. If this propagation is omitted or implemented inconsistently, +column names may be lost, reset, or become inconsistent across operations. +This issue becomes particularly apparent for operations that repartition frames, +such as `cbind` and `rbind`. + +## Initial Architecture + +A FrameObject acts as the runtime representation of a logical frame, while the underlying +frame data may be represented by one or more `FrameBlock`s. + +During execution, several operations create, combine, replace, or select +subsets of `FrameBlock`s. The investigated operations included: + +* `cbind` +* `rbind` +* `leftIindexing` +* `slice` + +Because column names were stored in individual `FrameBlock`s +rather than managed centrally at runtime, each of these execution +paths had to preserve or reconstruct the relevant names explicitly. + + +```mermaid +flowchart TB + + subgraph FO[FrameObject] + direction LR + + subgraph FB1[FrameBlock 1] + D1[data] + M1[column names and schema] + end + + subgraph FB2[FrameBlock 2] + D2[data] + M2[column names and schema] + end + + subgraph FB3[FrameBlock 3] + D3[data] + M3[column names and schema] + end + end +``` + +## Analysis + +To identify the root cause of the metadata propagation issues, several aspects of the existing +implementation were investigated. This included experimental propagation tests, an analysis +of the relevant system components, and an examination of the metadata flow throughout the +frame processing pipeline. The methodology and findings of this analysis form the basis for the design +decisions presented in the following section. + +### Analysis Methodology + +The analysis was conducted in two distinct phases. + +The first phase focused on understanding the existing metadata handling by tracing a +typical frame processing workflow using the `cbind` operation as a representative example. +The complete execution path was analyzed step by step with the help of a debugger, +allowing the involved components and the metadata flow to be examined in detail. +Although no metadata loss was observed during this initial analysis, several instructions +and components were identified as potential sources of inconsistent metadata propagation. + +The second phase aimed to verify these observations experimentally. +A series of propagation tests was developed to exercise the identified +execution paths under different conditions, particularly in scenarios where frames +are partitioned into multiple `FrameBlock`s. These tests confirmed the suspected +metadata propagation issues and provided the basis for the subsequent redesign. + +### Metadata Flow + +The metadata flow was analyzed by tracing representative frame operations through the +runtime execution pipeline. Starting from the `ExecutionContext`, the associated +`FrameObject` and its underlying `FrameBlock` instances were inspected to determine +where column names were stored, accessed, and propagated. + +The analysis showed that runtime instructions obtain frame variables through the +`ExecutionContext`, while many frame operations are ultimately executed directly on +`FrameBlock` instances. Since column names were stored at block level, newly created +blocks had to receive the corresponding metadata explicitly. + +This became particularly relevant for operations that create, combine, partition, or +replace `FrameBlock` instances. If the respective instruction did not copy or restore +the column names, the resulting frame could lose its metadata even though the underlying +data remained correct. + +In the original design, the propagation of column names along this path depended on the +individual instruction implementation. This instruction-specific handling was identified +as a major source of inconsistent metadata behavior. + + +### Relevant Components + +The execution trace identified a considerably larger number of involved classes. +The following components were selected for detailed investigation because +they either create new `FrameObject`s, manipulate `FrameBlock`s directly, or +are responsible for metadata propagation between runtime objects. + +| Component | Responsibility | Why Investigated | Observations | +|-----------|----------------|------------------|--------------| +| `FrameBlock` | Original storage location for frame metadata, including column names. | Investigated to understand how column names are stored, accessed, and propagated. In particular, the relationship between `FrameBlock` and `FrameObject` was analyzed to evaluate possible approaches for centralizing metadata management. | The analysis showed that `FrameBlock`s do not maintain a reference to their owning `FrameObject`. Consequently, metadata cannot be propagated directly from a `FrameBlock` to its corresponding `FrameObject`. Furthermore, several instructions operate directly on standalone `FrameBlock` instances without access to an `ExecutionContext`, making centralized metadata management more challenging. | +| `FrameObject` | Runtime representation of a frame and proposed central location for runtime metadata. | Investigated to determine whether it could serve as the authoritative source for metadata and how metadata could be synchronized with the associated `FrameBlock`s. | Since `FrameObject` represents the runtime abstraction of a frame, it provides a suitable location for storing frame-level metadata such as column names. Additional accessor methods can be introduced without significantly affecting the existing architecture. | +| `ExecutionContext` | Manages runtime variables and provides access to `FrameObject` instances during instruction execution. | Investigated to understand how runtime instructions obtain `FrameObject`s and whether this access path could be used to support centralized metadata propagation. | The `ExecutionContext` provides reliable access to the corresponding `FrameObject` during instruction execution. However, it is only available within runtime instructions, whereas many lower-level operations manipulate `FrameBlock`s directly without access to an `ExecutionContext`. | + +### Relevant Methods + +Based on the execution trace, the following methods were identified as particularly relevant for the +column-name metadata analysis. They represent important execution points where `FrameObject`s or +`FrameBlock`s are created, transformed, combined, or synchronized. + +| Class | Method | Purpose | Why Investigated | Observations | +| --------------------------- | --------------------------- | ------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| `BuiltinNaryCPInstruction` | `processInstruction()` | Executes n-ary built-in operations in CP, including `cbind` | Creates a new output frame during CP column-wise concatenation | Schema and column names must be propagated explicitly to the output. | +| `BuiltinNarySPInstruction` | `processInstruction()` | Executes distributed n-ary built-in operations, including `cbind` | Implements the Spark execution path for column-wise concatenation | Column names must be merged and synchronized with the output `FrameObject`. | +| `FrameAppendRSPInstruction` | `processInstruction()` | Executes distributed frame append operations for both `cbind` and `rbind` | Handles an additional Spark append path that creates or combines distributed `FrameBlock`s | For `cbind`, column names from both inputs are concatenated; for `rbind`, the column names of the left input are preserved. The resulting names must be assigned to both the output `FrameObject` and its distributed `FrameBlock`s. | +| `ReblockSPInstruction` | `processInstruction()` | Reblocks frames for distributed execution | May create a new distributed block representation of an existing frame | Column names must remain available after reblocking. | +| `CSVReblockSPInstruction` | `processInstruction()` | Converts CSV input into distributed frame blocks | Acts as an entry point for frames imported from CSV | Column names obtained from the CSV header must be transferred to the output runtime representation. | +| `ExecutionContext` | `createFrameObject()` | Creates new `FrameObject` instances | Central creation path for runtime frame objects | Column names must be initialized when a `FrameObject` is created from an existing `FrameBlock`. | +| `MLContextConversionUtil` | `frameBlockToFrameObject()` | Converts a `FrameBlock` into a `FrameObject` | Represents an additional conversion path into the runtime representation | Existing column names must be preserved during the conversion. | +| `DecodeMatrix` | `execute()` | Creates frames by decoding encoded matrices | Represents an additional frame creation path | Column-name metadata must be initialized or propagated consistently for the resulting frame. | + + +### Propagation Tests + +The metadata propagation behavior was evaluated using a dedicated set of propagation tests. +The selected operations (`cbind`, `rbind`, `leftIndexing`, and `slice`) were chosen because +they represent common frame transformations that either create new `FrameBlock`s or operate +on subsets of existing frames. Consequently, these operations were considered particularly +likely to expose inconsistencies in metadata propagation. + +Each test was executed with progressively increasing frame sizes (10, 100, 1000, and 2500 rows) +in both CP and Spark execution modes. + +In the propagation test setup, larger inputs caused disproportionate execution times and memory pressure. +Therefore, 2500 rows were selected as the largest stable automated test configuration. + +The gradual increase in input size served two purposes. +First, it verified that metadata propagation behaved consistently for small and medium-sized +frames. Second, and more importantly, larger inputs were verified to produce multiple `FrameBlock`s +in the investigated execution paths. This allowed the propagation behavior to be observed under realistic +distributed execution conditions, where metadata synchronization between multiple blocks +becomes necessary. + +For every test case, the frame was assigned a predefined set of column names before the +respective operation was executed. The resulting frame was then inspected to verify whether +the original column names were preserved correctly. By comparing the behavior across different +operations, execution modes, and frame sizes, it was possible to identify the execution paths +where metadata propagation was incomplete or inconsistent. + +### Key Findings + +The analysis resulted in the following observations: + +1. Metadata is stored within individual `FrameBlock`s. +2. `FrameBlock`s do not maintain a reference to their owning `FrameObject`. +3. Runtime instructions obtain frames through the `ExecutionContext`. +4. Numerous low-level operations manipulate standalone `FrameBlock`s directly. +5. Metadata propagation is implemented individually across multiple instructions. +6. The current architecture lacks a single authoritative source for runtime metadata. + +## Solution Development + +Based on the findings of the analysis, several design alternatives were evaluated. +The following sections describe the objectives of the redesign, discuss the considered +approaches, and justify the final design decisions. + +### Objectives + +The redesigned metadata handling should adhere to the following design goals. + +#### I. Establish an Authoritative Runtime Source for Column Names + +Column names should have one clearly defined authoritative source at +runtime. Although block-level copies may remain for backward compatibility, +runtime instructions should use the `FrameObject` as the primary source. +This reduces the risk of inconsistent representations and avoids relying exclusively +on instruction-specific propagation between individual `FrameBlock`s. + + +#### II. Preserve Column Names Across Distributed Frame Operations + +Column names should remain consistent regardless of the execution mode +or the operations applied to a frame. In particular, metadata must be +preserved when frames are partitioned into multiple `FrameBlock`s during +operations such as `cbind` and `rbind`. + +#### III. Maintain Backward Compatibility with Existing `FrameBlock`-Based Components + +The redesigned metadata handling should integrate seamlessly with the existing SystemDS +architecture. Components that currently rely solely on `FrameBlock` should continue to function +, minimizing the impact on the existing codebase while allowing a gradual migration to the new design. + +### Design Alternatives + +During the analysis process, a total of three possible approaches emerged. + +##### I. MetaData as the Central Source +The first alternative was to extend the existing `MetaData` class to store column names +in addition to frame characteristics. This would consolidate all frame-related metadata +within a single dedicated structure. + +However, the `MetaData` class is primarily intended to store static properties such as +dimensions, block sizes, and file format information. Since column names represent mutable +runtime metadata that may change during execution, this approach would require extending +the responsibilities of the `MetaData` abstraction beyond its original purpose. + +##### II. FrameObject as the Central Source +The second approach was to store column names directly in the `FrameObject`. + +Since the `FrameObject` already represents the runtime state of a frame and manages its +cached data, it provides a natural location for mutable frame-level metadata. Furthermore, +metadata propagation can be performed independently of individual `FrameBlock` instances, +avoiding the inconsistencies observed in the previous implementation. + +##### III. Complete Migration +A third option would have been to remove column names entirely from the `FrameBlock` +and store them exclusively in the `FrameObject`. + +While this would eliminate duplicated metadata completely, it would also require extensive +changes to existing components that intentionally operate directly on standalone +`FrameBlock` instances, including low-level I/O and legacy APIs. + +### Design Decisions + +Based on the analysis, **the second approach was selected**. + +The `FrameObject` was introduced as the authoritative source for column names. Since +runtime instructions already operate on `FrameObject` instances through the +`ExecutionContext`, metadata propagation can be performed consistently during frame +operations without relying on individual `FrameBlock` instances. + +The existing representation of column names inside the `FrameBlock` was intentionally +retained. This preserves compatibility with components that explicitly operate on +standalone `FrameBlock` objects while allowing runtime metadata management to be handled +centrally through the `FrameObject`. + +This design therefore separates the runtime management of column names from +their block-level representation. + +### Solution Implementation + +After the conceptual design had been finalized, the required changes were implemented +across the affected SystemDS components. The following sections summarize the modified +classes and explain how metadata propagation was adapted. + +#### Implementation Overview + +| Class | Method | Purpose | Changes | +| :----------------------------------- | :---------------------------------- | :---------------------------------------------------- | :------------------------------------------------------------------------------------------------- | +| `FrameObject` | - | Runtime representation of a frame. | Added runtime storage for column names. | +| `FrameObject` | `getColumnNames()` | Returns all column names. | Implemented runtime column-name retrieval. | +| `FrameObject` | `getColumnNames(int cl, int cu)` | Returns a subset of column names. | Implemented partial column-name retrieval. | +| `FrameObject` | `setColumnNames(String[] colnames)` | Updates runtime column names. | Implemented runtime column-name assignment. | +| `FrameObject` | `mergeColumnNames(FrameObject fo)` | Merges column names of two `FrameObject`s. | Implemented column-name merging for frame concatenation. | +| `FrameObject` | `readBlobFromHDFS()` | Loads frame data from HDFS. | Initialized runtime column names from imported CSV and Parquet metadata if not already present. | +| `ExecutionContext` | `createFrameObject()` | Creates new `FrameObject` instances. | Preserved column names when creating a `FrameObject` from a `FrameBlock`. | +| `FrameReaderTextCSV` | `readColumnNamesFromHDFS()` | Reads CSV frame headers. | Added dedicated CSV header parsing to extract column names during import. | +| `BuiltinNarySPInstruction` | `processInstruction()` | Executes distributed n-ary frame operations. | Added column-name propagation for `cbind` by merging the names of all input `FrameObject`s. | +| `FrameAppendRSPInstruction` | `processInstruction()` | Executes distributed frame append operations. | Added column-name propagation for `cbind` by concatenating the names of both input `FrameObject`s. | +| `FrameIndexingSPInstruction` | `processInstruction()` | Executes distributed frame indexing operations. | Propagated the corresponding subset of column names to the output `FrameObject`. | +| `CSVReblockSPInstruction` | `processInstruction()` | Reblocks CSV frames for distributed execution. | Propagated column names from the input to the output `FrameObject`. | +| `ReblockSPInstruction` | `processInstruction()` | Reblocks frames for distributed execution. | Propagated column names from the input to the output `FrameObject`. | +| `PreparedScript` | `setFrame()` | Registers input frames for execution. | Propagated column names when creating a new input `FrameObject`. | +| `MLContextConversionUtil` | `frameBlockToFrameObject()` | Converts `FrameBlock`s to `FrameObject`s. | Preserved column names during `FrameBlock`-to-`FrameObject` conversion. | +| `ParameterizedBuiltinFEDInstruction` | `processInstruction()` | Executes federated parameterized built-in operations. | Propagated schema and column names to decoded `FrameObject`s. | + +## Validation +The redesigned metadata handling was validated using the previously developed +propagation test suite. The tests were executed in both CP and Spark execution +modes and covered the operations cbind, rbind, leftIndexing, and slice using +frame sizes ranging from 10 to 2500 rows. For every test case, predefined +column names were assigned before the operation and verified afterwards. +Following the implementation, all propagation tests completed successfully, +demonstrating that column names are preserved consistently across the investigated +execution paths and frame partitioning scenarios. + +## Future Work + +While the proposed solution centralizes the runtime management of column names without +breaking the existing architecture, a more fundamental redesign could be considered in +the future. Such an approach would remove the remaining duplication by establishing a +single representation of column names throughout the frame infrastructure and limiting +the responsibility of FrameBlock to storing block-level data. + +However, this would introduce significant breaking changes and require +substantial modifications to the existing data-handling architecture. +In particular, components and instructions that currently operate directly +on FrameBlock instances would need to be adapted to the new design. +Consequently, such a redesign should only be considered after carefully evaluating +its impact on compatibility, maintainability, and the existing runtime infrastructure. diff --git a/docs/site/dml-language-reference.md b/docs/site/dml-language-reference.md index 264b3c6a2b1..6a26c4caaac 100644 --- a/docs/site/dml-language-reference.md +++ b/docs/site/dml-language-reference.md @@ -2068,10 +2068,12 @@ The following example uses transformapply() with the input matrix a **Table F5**: Frame processing built-in functions -Function | Description | Parameters | Example --------- | ----------- | ---------- | ------- -map() | It will execute the given lambda expression on a frame (cell, row or column wise). | Input: (X <frame>, y <String>, \[margin <int>\])
Output: <frame>.
X is a frame and
y is a String containing the lambda expression to be executed on frame X.
margin - how to apply the lambda expression (0 indicates each cell, 1 - rows, 2 - columns). Output matrix dimensions are always equal to the input. | [map](#map) -tokenize() | Transforms a frame to tokenized frame using specification. Tokenization is valid only for string columns. | Input:
target = <frame>
spec = <json specification>
Outputs: <matrix>, <frame> | [tokenize](#tokenize) +Function | Description | Parameters | Example +-------- |-----------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| ------- +map() | It will execute the given lambda expression on a frame (cell, row or column wise). | Input: (X <frame>, y <String>, \[margin <int>\])
Output: <frame>.
X is a frame and
y is a String containing the lambda expression to be executed on frame X.
margin - how to apply the lambda expression (0 indicates each cell, 1 - rows, 2 - columns). Output matrix dimensions are always equal to the input. | [map](#map) +tokenize() | Transforms a frame to tokenized frame using specification. Tokenization is valid only for string columns. | Input:
target = <frame>
spec = <json specification>
Outputs: <matrix>, <frame> | [tokenize](#tokenize) +getNames() | Returns the column names of a frame as a single-row frame. | Input: X <frame>
Output: <frame> | N = getNames(X) +setNames() | Sets the column names of a frame from a single-row frame containing string values. | Input:
X = <frame>
N = <frame>
Output:<frame> | Y = setNames(X, N) #### map diff --git a/src/main/java/org/apache/sysds/api/jmlc/PreparedScript.java b/src/main/java/org/apache/sysds/api/jmlc/PreparedScript.java index 31bb7457227..ae50c2d18f6 100644 --- a/src/main/java/org/apache/sysds/api/jmlc/PreparedScript.java +++ b/src/main/java/org/apache/sysds/api/jmlc/PreparedScript.java @@ -397,6 +397,8 @@ public void setFrame(String varname, FrameBlock frame, boolean reuse) { MetaDataFormat meta = new MetaDataFormat(mc, FileFormat.BINARY); FrameObject fo = new FrameObject(OptimizerUtils.getUniqueTempFileName(), meta); fo.acquireModify(frame); + fo.setSchema(frame.getSchema()); + fo.setColumnNames(frame.getColumnNames()); fo.release(); //put create matrix wrapper into symbol table diff --git a/src/main/java/org/apache/sysds/api/mlcontext/MLContextConversionUtil.java b/src/main/java/org/apache/sysds/api/mlcontext/MLContextConversionUtil.java index 7e903428ae3..ec207893bd1 100644 --- a/src/main/java/org/apache/sysds/api/mlcontext/MLContextConversionUtil.java +++ b/src/main/java/org/apache/sysds/api/mlcontext/MLContextConversionUtil.java @@ -189,6 +189,7 @@ public static FrameObject frameBlockToFrameObject(String variableName, FrameBloc FrameObject frameObject = new FrameObject(OptimizerUtils.getUniqueTempFileName(), mtd, frameMetadata.getFrameSchema().getSchema().toArray(new ValueType[0])); frameObject.acquireModify(frameBlock); + frameObject.setColumnNames(frameBlock.getColumnNames()); frameObject.release(); return frameObject; } catch (DMLRuntimeException e) { @@ -299,6 +300,7 @@ public static FrameObject binaryBlocksToFrameObject(JavaPairRDD", InstructionType.Binary), MAPGE("map>=", InstructionType.Binary), diff --git a/src/main/java/org/apache/sysds/common/Types.java b/src/main/java/org/apache/sysds/common/Types.java index 624c9eed3c6..b8614ea0a0c 100644 --- a/src/main/java/org/apache/sysds/common/Types.java +++ b/src/main/java/org/apache/sysds/common/Types.java @@ -641,7 +641,8 @@ public enum OpOp2 { MINUS1_MULT(false), //1-X*Y GET_CATEGORICAL_MASK(false), // get transformation mask QUANTIZE_COMPRESS(false), //quantization-fused compression - UNION_DISTINCT(false); + UNION_DISTINCT(false), + SET_COLNAMES(false); private final boolean _validOuter; diff --git a/src/main/java/org/apache/sysds/parser/BuiltinFunctionExpression.java b/src/main/java/org/apache/sysds/parser/BuiltinFunctionExpression.java index ab0c7993b4e..71e9820deea 100644 --- a/src/main/java/org/apache/sysds/parser/BuiltinFunctionExpression.java +++ b/src/main/java/org/apache/sysds/parser/BuiltinFunctionExpression.java @@ -1095,6 +1095,7 @@ else if( getAllExpr().length == 2 ) { //binary case TYPEOF: case DETECTSCHEMA: case COLNAMES: + case GET_NAMES: checkNumParameters(1); checkMatrixFrameParam(getFirstExpr()); output.setDataType(DataType.FRAME); @@ -1102,6 +1103,26 @@ else if( getAllExpr().length == 2 ) { //binary output.setBlocksize (id.getBlocksize()); output.setValueType(ValueType.STRING); break; + case SET_NAMES: + //check if we use 2 parameters (Frame on which nemas are set and vector for names) + checkNumParameters(2); + + // check if first paramters is a frame + checkMatrixFrameParam(getFirstExpr()); + + // check if second paramters is a vector 1xn Frame + checkMatrixFrameParam(getSecondExpr()); + + //output should be a frame + output.setDataType(DataType.FRAME); + + + checkMatrixFrameParam(getFirstExpr()); + output.setDataType(DataType.FRAME); + output.setDimensions(id.getDim1(), id.getDim2()); + output.setBlocksize (id.getBlocksize()); + output.setDataType(DataType.FRAME); + break; case CAST_AS_FRAME: // operation as.frame // overloaded to take either one argument or 2 where second is column names diff --git a/src/main/java/org/apache/sysds/parser/DMLTranslator.java b/src/main/java/org/apache/sysds/parser/DMLTranslator.java index a8e1667d049..e341139fb40 100644 --- a/src/main/java/org/apache/sysds/parser/DMLTranslator.java +++ b/src/main/java/org/apache/sysds/parser/DMLTranslator.java @@ -2763,6 +2763,34 @@ else if ( in.length == 2 ) case TYPEOF: case DET: case DETECTSCHEMA: + currBuiltinOp = new UnaryOp( + target.getName(), + target.getDataType(), + target.getValueType(), + OpOp1.valueOf(source.getOpCode().name()), + expr + ); + break; + + case SET_NAMES: + currBuiltinOp = new BinaryOp( + target.getName(), + target.getDataType(), + target.getValueType(), + OpOp2.SET_COLNAMES, + expr, + expr2 + ); + break; + + case GET_NAMES: + currBuiltinOp = new UnaryOp( + target.getName(), + target.getDataType(), + target.getValueType(), + OpOp1.COLNAMES, expr + ); + break; case COLNAMES: currBuiltinOp = new UnaryOp(target.getName(), target.getDataType(), target.getValueType(), OpOp1.valueOf(source.getOpCode().name()), expr); diff --git a/src/main/java/org/apache/sysds/runtime/controlprogram/caching/FrameObject.java b/src/main/java/org/apache/sysds/runtime/controlprogram/caching/FrameObject.java index 87d14dbf87e..e2530fc1d23 100644 --- a/src/main/java/org/apache/sysds/runtime/controlprogram/caching/FrameObject.java +++ b/src/main/java/org/apache/sysds/runtime/controlprogram/caching/FrameObject.java @@ -38,10 +38,7 @@ import org.apache.sysds.runtime.instructions.ooc.OOCStream; import org.apache.sysds.runtime.instructions.spark.data.IndexedMatrixValue; import org.apache.sysds.runtime.instructions.spark.data.RDDObject; -import org.apache.sysds.runtime.io.FileFormatProperties; -import org.apache.sysds.runtime.io.FrameReaderFactory; -import org.apache.sysds.runtime.io.FrameWriter; -import org.apache.sysds.runtime.io.FrameWriterFactory; +import org.apache.sysds.runtime.io.*; import org.apache.sysds.runtime.lineage.LineageItem; import org.apache.sysds.runtime.lineage.LineageRecomputeUtils; import org.apache.sysds.runtime.meta.DataCharacteristics; @@ -61,7 +58,9 @@ public class FrameObject extends CacheableData private static final long serialVersionUID = 1755082174281927785L; private ValueType[] _schema = null; - + + private String[] _colnames = null; + protected FrameObject() { super(DataType.FRAME, ValueType.STRING); } @@ -85,8 +84,8 @@ public FrameObject(String fname, MetaData meta, ValueType[] schema) { } /** - * Copy constructor that copies meta data but NO data. - * + * Copy constructor that copies meta data and column names but NO data. + * * @param fo frame object */ public FrameObject(FrameObject fo) { @@ -94,9 +93,10 @@ public FrameObject(FrameObject fo) { MetaDataFormat metaOld = (MetaDataFormat) fo.getMetaData(); _metaData = new MetaDataFormat( - new MatrixCharacteristics(metaOld.getDataCharacteristics()), - metaOld.getFileFormat()); - _schema = fo._schema.clone(); + new MatrixCharacteristics(metaOld.getDataCharacteristics()), + metaOld.getFileFormat()); + _schema = fo._schema != null ? fo._schema.clone() : null; + _colnames = fo._colnames != null ? fo._colnames.clone() : null; } @Override @@ -112,8 +112,8 @@ public ValueType[] getSchema() { * @return schema of value types */ public ValueType[] getSchema(int cl, int cu) { - return (_schema!=null && _schema.length>cu) ? Arrays.copyOfRange(_schema, cl, cu+1) : - UtilFunctions.nCopies(cu-cl+1, ValueType.STRING); + return (_schema != null && _schema.length > cu) ? Arrays.copyOfRange(_schema, cl, cu + 1) : + UtilFunctions.nCopies(cu - cl + 1, ValueType.STRING); } /** @@ -125,10 +125,14 @@ public ValueType[] getSchema(int cl, int cu) { */ public ValueType[] mergeSchemas(FrameObject fo) { return ArrayUtils.addAll( - (_schema!=null) ? _schema : UtilFunctions.nCopies((int)getNumColumns(), ValueType.STRING), - (fo._schema!=null) ? fo._schema : UtilFunctions.nCopies((int)fo.getNumColumns(), ValueType.STRING)); - } - + (_schema != null) ? _schema : UtilFunctions.nCopies((int) getNumColumns(), ValueType.STRING), + (fo._schema != null) ? fo._schema : UtilFunctions.nCopies((int) fo.getNumColumns(), ValueType.STRING)); + } + + /** + * + * @param schema + */ public void setSchema(String schema) { if( schema.equals("*") ) { //populate default schema @@ -154,17 +158,44 @@ public static ValueType[] parseSchema(String schema) { public void setSchema(ValueType[] schema) { _schema = schema; } - + + public String[] getColumnNames() { + return _colnames; + } + + /** + * Obtain column names + * + * @param cl column lower bound, inclusive + * @param cu column upper bound, inclusive + * @return column names + */ + public String[] getColumnNames(int cl, int cu) { + return (_colnames != null && _colnames.length > cu) + ? Arrays.copyOfRange(_colnames, cl, cu + 1) + : FrameBlock.createColNames(cu - cl + 1); + } + + /** + * + * @param colNames + */ + public void setColumnNames(String[] colNames) { + _colnames = colNames != null + ? colNames.clone() + : null; + } + @Override public void refreshMetaData() { - if ( _data == null || _metaData ==null ) //refresh only for existing data - throw new DMLRuntimeException("Cannot refresh meta data because there is no data or meta data. "); + if (_data == null || _metaData == null) //refresh only for existing data + throw new DMLRuntimeException("Cannot refresh meta data because there is no data or meta data. "); //update matrix characteristics DataCharacteristics dc = _metaData.getDataCharacteristics(); - dc.setDimension( _data.getNumRows(),_data.getNumColumns() ); - dc.setNonZeros(_data.getNumRows()*_data.getNumColumns()); - + dc.setDimension(_data.getNumRows(), _data.getNumColumns()); + dc.setNonZeros(_data.getNumRows() * _data.getNumColumns()); + //update schema information _schema = _data.getSchema(); } @@ -208,15 +239,29 @@ protected FrameBlock readBlobFromHDFS(String fname, long[] dims) throws IOExcept if(data == null) throw new IOException("Unable to load frame from file: " + fname); + FileFormat format = iimd.getFileFormat(); + //Delta and CSV discover dimensions (and Delta also schema) at read time, so //refresh the cached metadata to reflect the materialized frame block. - if(iimd.getFileFormat() == FileFormat.CSV || iimd.getFileFormat() == FileFormat.DELTA) { + if(format == FileFormat.CSV || format == FileFormat.DELTA) { _metaData = _metaData instanceof MetaDataFormat ? new MetaDataFormat(data.getDataCharacteristics(), - iimd.getFileFormat()) : new MetaData(data.getDataCharacteristics()); - if(iimd.getFileFormat() == FileFormat.DELTA) + format) : new MetaData(data.getDataCharacteristics()); + + if(format == FileFormat.DELTA) _schema = data.getSchema(); } + if (format == FileFormat.PARQUET) + _schema = data.getSchema(); + + if (_colnames == null && (format == FileFormat.CSV || format == FileFormat.PARQUET)) { + String[] columnNames = data.getColumnNames(); + + if (columnNames != null) { + setColumnNames(columnNames); + } + } + return data; } diff --git a/src/main/java/org/apache/sysds/runtime/controlprogram/context/ExecutionContext.java b/src/main/java/org/apache/sysds/runtime/controlprogram/context/ExecutionContext.java index 67cda352a73..3df8b594d58 100644 --- a/src/main/java/org/apache/sysds/runtime/controlprogram/context/ExecutionContext.java +++ b/src/main/java/org/apache/sysds/runtime/controlprogram/context/ExecutionContext.java @@ -703,6 +703,7 @@ public static FrameObject createFrameObject(FrameBlock fb) { ret.acquireModify(fb); ret.setMetaData(new MetaDataFormat(new MatrixCharacteristics( fb.getNumRows(), fb.getNumColumns()), FileFormat.BINARY)); + ret.setColumnNames(fb.getColumnNames()); ret.release(); return ret; } diff --git a/src/main/java/org/apache/sysds/runtime/frame/data/FrameBlock.java b/src/main/java/org/apache/sysds/runtime/frame/data/FrameBlock.java index 63cadb43cf4..668e04dce94 100644 --- a/src/main/java/org/apache/sysds/runtime/frame/data/FrameBlock.java +++ b/src/main/java/org/apache/sysds/runtime/frame/data/FrameBlock.java @@ -85,31 +85,49 @@ public class FrameBlock implements CacheBlock, Externalizable { private static final Log LOG = LogFactory.getLog(FrameBlock.class.getName()); private static final long serialVersionUID = -3993450030207130665L; private static final IDSequence CLASS_ID = new IDSequence(); - /** Buffer size variable: 1M elements, size of default matrix block */ + /** + * Buffer size variable: 1M elements, size of default matrix block + */ public static final int BUFFER_SIZE = 1 * 1000 * 1000; - /** If debugging is enabled for the FrameBlocks in stable state */ + /** + * If debugging is enabled for the FrameBlocks in stable state + */ public static boolean debug = false; - /** The schema of the data frame as an ordered list of value types */ + /** + * The schema of the data frame as an ordered list of value types + */ private ValueType[] _schema = null; - /** The column names of the data frame as an ordered list of strings, allocated on-demand */ + /** + * The column names of the data frame as an ordered list of strings, allocated on-demand + */ private String[] _colnames = null; - /** The column metadata */ + /** + * The column metadata + */ private ColumnMetadata[] _colmeta = null; - /** The data frame data as an ordered list of columns */ + /** + * The data frame data as an ordered list of columns + */ private Array[] _coldata = null; - /** Locks on the columns not tied to the columns objects. */ + /** + * Locks on the columns not tied to the columns objects. + */ private SoftReference _columnLocks = null; - /** Materialized number of rows in this FrameBlock */ + /** + * Materialized number of rows in this FrameBlock + */ private int _nRow = 0; - /** Cached size in memory to avoid repeated scans of string columns */ + /** + * Cached size in memory to avoid repeated scans of string columns + */ private long _msize = -1; public FrameBlock() { @@ -156,7 +174,7 @@ public FrameBlock(ValueType[] schema, String[][] data) { /** * FrameBlock constructor with constant - * + * * @param schema The schema to allocate (also specifying number of columns) * @param constant The constant to allocate in all cells * @param nRow the number of rows @@ -165,29 +183,29 @@ public FrameBlock(ValueType[] schema, String constant, int nRow) { this(); // allocate the values. _nRow = nRow; - for(int i = 0; i < schema.length; i++) + for (int i = 0; i < schema.length; i++) appendColumn(ArrayFactory.allocate(schema[i], nRow, constant)); } /** * allocate a FrameBlock with the given data arrays. - * + *

* The data is in row major, making the first dimension number of rows. second number of columns. - * + * * @param schema the schema to allocate * @param names The names of the column * @param data The data. */ public FrameBlock(ValueType[] schema, String[] names, String[][] data) { _schema = schema; - if(names != null) { + if (names != null) { _colnames = names; - if(schema.length != names.length) + if (schema.length != names.length) throw new DMLRuntimeException("Invalid FrameBlock construction, invalid schema and names combination"); } ensureAllocateMeta(); - if(data != null) { - for(int i = 0; i < data.length; i++) + if (data != null) { + for (int i = 0; i < data.length; i++) appendRow(data[i]); } } @@ -202,12 +220,12 @@ public FrameBlock(ValueType[] schema, String[] colNames, ColumnMetadata[] meta, /** * Create a FrameBlock containing columns of the specified arrays - * + * * @param data The column data contained */ public FrameBlock(Array[] data) { _schema = new ValueType[data.length]; - for(int i = 0; i < data.length; i++) + for (int i = 0; i < data.length; i++) _schema[i] = data[i].getValueType(); _colnames = null; @@ -215,24 +233,24 @@ public FrameBlock(Array[] data) { _coldata = data; _nRow = data[0].size(); - if(debug) { - for(int i = 0; i < data.length; i++) { - if(data[i].size() != getNumRows()) + if (debug) { + for (int i = 0; i < data.length; i++) { + if (data[i].size() != getNumRows()) throw new DMLRuntimeException("Invalid Frame allocation with different size arrays " - + data[i].size() + " vs " + getNumRows()); + + data[i].size() + " vs " + getNumRows()); } } } /** * Create a FrameBlock containing columns of the specified arrays and names - * + * * @param data The column data contained * @param colnames The column names of the contained columns */ public FrameBlock(Array[] data, String[] colnames) { _schema = new ValueType[data.length]; - for(int i = 0; i < data.length; i++) + for (int i = 0; i < data.length; i++) _schema[i] = data[i].getValueType(); _colnames = colnames; @@ -240,11 +258,11 @@ public FrameBlock(Array[] data, String[] colnames) { _coldata = data; _nRow = data[0].size(); - if(debug) { - for(int i = 0; i < data.length; i++) { - if(data[i].size() != getNumRows()) + if (debug) { + for (int i = 0; i < data.length; i++) { + if (data[i].size() != getNumRows()) throw new DMLRuntimeException("Invalid Frame allocation with different size arrays " - + data[i].size() + " vs " + getNumRows()); + + data[i].size() + " vs " + getNumRows()); } } } @@ -273,7 +291,7 @@ public double getDoubleNaN(int r, int c) { public String getString(int r, int c) { Object o = get(r, c); String s = (o == null) ? null : o.toString(); - if(s != null && s.isEmpty()) + if (s != null && s.isEmpty()) return null; return s; } @@ -333,7 +351,7 @@ public FrameBlock getColumnNamesAsFrame() { * @return array of column names */ public String[] getColumnNames(boolean alloc) { - if(_colnames == null && alloc) + if (_colnames == null && alloc) _colnames = createColNames(getNumColumns()); return _colnames; } @@ -345,7 +363,7 @@ public String[] getColumnNames(boolean alloc) { * @return column name */ public String getColumnName(int c) { - if(_colnames == null) + if (_colnames == null) _colnames = createColNames(getNumColumns()); return _colnames[c]; } @@ -355,7 +373,7 @@ public void setColumnNames(String[] colnames) { } public void setColumnName(int index, String name) { - if(_colnames == null) + if (_colnames == null) _colnames = createColNames(getNumColumns()); _colnames[index] = name; } @@ -374,7 +392,7 @@ public Array[] getColumns() { public boolean isColumnMetadataDefault() { boolean ret = true; - for(int j = 0; j < getNumColumns() && ret; j++) + for (int j = 0; j < getNumColumns() && ret; j++) ret &= isColumnMetadataDefault(j); return ret; } @@ -384,7 +402,13 @@ public boolean isColumnMetadataDefault(int c) { } public void setColumnMetadata(ColumnMetadata[] colmeta) { - System.arraycopy(colmeta, 0, _colmeta, 0, _colmeta.length); + if (colmeta == null) + return; + + for (int i = 0; i < _colmeta.length; i++) + _colmeta[i] = colmeta[i] != null + ? new ColumnMetadata(colmeta[i]) + : new ColumnMetadata(); } public void setColumnMetadata(int c, ColumnMetadata colmeta) { @@ -398,7 +422,7 @@ public void setColumnMetadata(int c, ColumnMetadata colmeta) { */ public Map getColumnNameIDMap() { Map ret = new HashMap<>(); - for(int j = 0; j < getNumColumns(); j++) + for (int j = 0; j < getNumColumns(); j++) ret.put(getColumnName(j), j + 1); return ret; } @@ -415,46 +439,45 @@ public void ensureAllocatedColumns(int numRows) { // allocate column meta data if necessary ensureAllocateMeta(); // early abort if already allocated - if(_coldata != null && _schema.length == _coldata.length) { + if (_coldata != null && _schema.length == _coldata.length) { // handle special case that to few rows allocated - if(nRow < numRows) { + if (nRow < numRows) { String[] tmp = new String[getNumColumns()]; int len = numRows - nRow; // TODO: Add append N function. - for(int i = 0; i < len; i++) + for (int i = 0; i < len; i++) appendRow(tmp); } return; - } - else { + } else { // allocate columns if necessary _coldata = new Array[_schema.length]; - if(numRows > 0) - for(int j = 0; j < _schema.length; j++) + if (numRows > 0) + for (int j = 0; j < _schema.length; j++) _coldata[j] = ArrayFactory.allocate(_schema[j], numRows); _nRow = numRows; } } private void ensureAllocateMeta() { - if(_colmeta == null || _schema.length != _colmeta.length) { + if (_colmeta == null || _schema.length != _colmeta.length) { _colmeta = new ColumnMetadata[_schema.length]; - for(int j = 0; j < _schema.length; j++) + for (int j = 0; j < _schema.length; j++) _colmeta[j] = new ColumnMetadata(); } } /** * Checks for matching column sizes in case of existing columns. - * + *

* If the check parses the number of rows is reassigned to the given newLen * * @param newLen number of rows to compare with existing number of rows */ public void ensureColumnCompatibility(int newLen) { final int nRow = getNumRows(); - if(_coldata != null && _coldata.length > 0 && ((nRow == 0) || nRow != newLen)) { + if (_coldata != null && _coldata.length > 0 && ((nRow == 0) || nRow != newLen)) { throw new RuntimeException("Mismatch in number of rows: " + newLen + " (expected: " + nRow + ")"); } _nRow = newLen; @@ -466,7 +489,7 @@ public static String[] createColNames(int size) { public static String[] createColNames(int off, int size) { String[] ret = new String[size]; - for(int i = off + 1; i <= off + size; i++) + for (int i = off + 1; i <= off + size; i++) ret[i - off - 1] = createColName(i); return ret; } @@ -477,7 +500,7 @@ public static String createColName(int i) { public boolean isColNamesDefault() { boolean ret = (_colnames != null); - for(int j = 0; j < getNumColumns() && ret; j++) + for (int j = 0; j < getNumColumns() && ret; j++) ret &= isColNameDefault(j); return ret; } @@ -487,9 +510,9 @@ public boolean isColNameDefault(int i) { } public void recomputeColumnCardinality() { - for(int j = 0; j < getNumColumns(); j++) { + for (int j = 0; j < getNumColumns(); j++) { int card = 0; - for(int i = 0; i < getNumRows(); i++) + for (int i = 0; i < getNumRows(); i++) card += (get(i, j) != null) ? 1 : 0; _colmeta[j].setNumDistinct(card); } @@ -524,7 +547,7 @@ public void set(int r, int c, Object val) { /** * Sets the value in position (r,c), to the input string value, and at the individual arrays, convert to correct * type. - * + * * @param r row index * @param c column index * @param val value to set at specified position @@ -534,17 +557,17 @@ public void set(int r, int c, String val) { } public void reset(int nrow, boolean clearMeta) { - if(clearMeta) { + if (clearMeta) { _schema = null; _colnames = null; - if(_colmeta != null) { - for(int i = 0; i < _colmeta.length; i++) - if(!isColumnMetadataDefault(i)) + if (_colmeta != null) { + for (int i = 0; i < _colmeta.length; i++) + if (!isColumnMetadataDefault(i)) _colmeta[i] = new ColumnMetadata(); } } - if(_coldata != null) { - for(int i = 0; i < _coldata.length; i++) + if (_coldata != null) { + for (int i = 0; i < _coldata.length; i++) _coldata[i].reset(nrow); } _nRow = nrow; @@ -557,7 +580,8 @@ public void reset() { /** * Sets row at position r to the input array of objects, corresponding to the schema. - * @param r row index + * + * @param r row index * @param row array of objects */ public void setRow(int r, Object[] row) { @@ -568,24 +592,23 @@ public void setRow(int r, Object[] row) { /** * Append a row to the end of the data frame, where all row fields are boxed objects according to the schema. - * + *

* Append row should be avoided if possible. - * + * * @param row array of objects */ public void appendRow(Object[] row) { - if(row.length != _schema.length) + if (row.length != _schema.length) throw new DMLRuntimeException("Invalid number of values in rowAppend"); - if(_nRow == 0) { + if (_nRow == 0) { ensureAllocateMeta(); _coldata = new Array[_schema.length]; - for(int j = 0; j < _schema.length; j++) { + for (int j = 0; j < _schema.length; j++) { _coldata[j] = ArrayFactory.allocate(_schema[j], 1); _coldata[j].set(0, row[j]); } - } - else { - for(int j = 0; j < row.length; j++) + } else { + for (int j = 0; j < row.length; j++) _coldata[j].append(row[j]); } _nRow++; @@ -594,24 +617,23 @@ public void appendRow(Object[] row) { /** * Append a row to the end of the data frame, where all row fields are string encoded. - * + *

* Append row should be avoided if possible - * + * * @param row array of strings */ public void appendRow(String[] row) { - if(row.length != _schema.length) + if (row.length != _schema.length) throw new DMLRuntimeException("Invalid number of values in rowAppend"); - else if(_nRow == 0) { + else if (_nRow == 0) { ensureAllocateMeta(); _coldata = new Array[_schema.length]; - for(int j = 0; j < _schema.length; j++) { + for (int j = 0; j < _schema.length; j++) { _coldata[j] = ArrayFactory.allocate(_schema[j], 1); _coldata[j].set(0, row[j]); } - } - else { - for(int j = 0; j < row.length; j++) + } else { + for (int j = 0; j < row.length; j++) _coldata[j].append(row[j]); } _nRow++; @@ -692,11 +714,11 @@ public void appendColumn(double[] col) { /** * Append the metadata associated with adding a column. - * + * * @param vt The Value type */ private void appendColumnMetaData(ValueType vt) { - if(_colnames != null) + if (_colnames != null) _colnames = ArrayUtils.add(getColumnNames(), createColName(_colnames.length + 1)); _schema = ArrayUtils.add(_schema, vt); _colmeta = ArrayUtils.add(getColumnMetadata(), new ColumnMetadata()); @@ -714,11 +736,11 @@ public void appendColumns(double[][] cols) { boolean empty = (_schema == null); ValueType[] tmpSchema = UtilFunctions.nCopies(ncol, ValueType.FP64); Array[] tmpData = new Array[ncol]; - for(int j = 0; j < ncol; j++) + for (int j = 0; j < ncol; j++) tmpData[j] = ArrayFactory.create(cols[j]); _colnames = empty ? null : ArrayUtils.addAll(getColumnNames(), createColNames(getNumColumns(), ncol)); // before - // schema - // modification + // schema + // modification _schema = empty ? tmpSchema : ArrayUtils.addAll(_schema, tmpSchema); _coldata = empty ? tmpData : ArrayUtils.addAll(_coldata, tmpData); _nRow = cols[0].length; @@ -731,7 +753,7 @@ public static FrameBlock convertToFrameBlock(MatrixBlock mb, ValueType[] schema, /** * Add a column of already allocated Array type. - * + * * @param col column to add. */ public void appendColumn(Array col) { @@ -753,12 +775,11 @@ public Array getColumn(int c) { } public void setColumn(int c, Array column) { - if(_coldata == null) { + if (_coldata == null) { _coldata = new Array[getNumColumns()]; - if(column != null) + if (column != null) _nRow = column.size(); - } - else if(column != null && column.size() != _nRow) + } else if (column != null && column.size() != _nRow) throw new DMLRuntimeException("Invalid number of rows in set column"); _coldata[c] = column; _msize = -1; @@ -766,7 +787,7 @@ else if(column != null && column.size() != _nRow) /** * Appends a chunk of data to the end of a specified column. - * + * * @param c column index * @param chunk chunk of data to append */ @@ -788,8 +809,8 @@ public void appendColumnChunk(int c, Array chunk) { /** * Sets a chunk of data to a specified column, starting at the specified offset. - * - * @param c column index + * + * @param c column index * @param chunk chunk of data to set * @param offset offset position where it should set the chunk * @param colSize size of columns, in case columns aren't initialized yet @@ -822,14 +843,14 @@ public void write(DataOutput out) throws IOException { out.writeInt(getNumColumns()); out.writeBoolean(isDefaultMeta); // write columns (value type, data) - for(int j = 0; j < getNumColumns(); j++) { + for (int j = 0; j < getNumColumns(); j++) { final byte type = getTypeForIO(j); out.writeByte(type); - if(!isDefaultMeta) { + if (!isDefaultMeta) { out.writeUTF(getColumnName(j)); _colmeta[j].write(out); } - if(type > 0 && nRow > 0) // if allocated write column data + if (type > 0 && nRow > 0) // if allocated write column data _coldata[j].write(out); } } @@ -837,7 +858,7 @@ public void write(DataOutput out) throws IOException { private byte getTypeForIO(int col) { // ! +1 to allow reflecting around zero if not allocated byte type = (byte) (_schema[col].ordinal() + 1); - if(_coldata == null || _coldata[col] == null) + if (_coldata == null || _coldata[col] == null) type *= -1; // negative to indicate not allocated return type; } @@ -855,23 +876,22 @@ public void readFields(DataInput in) throws IOException { // allocate schema/meta data arrays _schema = (_schema != null && _schema.length == numCols) ? _schema : new ValueType[numCols]; _colnames = (_colnames != null && _colnames.length == numCols) ? _colnames : // if already allocated reuse - isDefaultMeta ? null : new String[numCols]; // if meta is default allocate on demand + isDefaultMeta ? null : new String[numCols]; // if meta is default allocate on demand _colmeta = (_colmeta != null && _colmeta.length == numCols) ? _colmeta : new ColumnMetadata[numCols]; _coldata = (_coldata != null && _coldata.length == numCols) ? _coldata : new Array[numCols]; - if(_nRow == 0) + if (_nRow == 0) _coldata = null; // read columns (value type, meta, data) - for(int j = 0; j < numCols; j++) { + for (int j = 0; j < numCols; j++) { byte type = in.readByte(); _schema[j] = interpretByteAsType(type); - if(!isDefaultMeta) { // If not default meta read in meta + if (!isDefaultMeta) { // If not default meta read in meta _colnames[j] = in.readUTF(); _colmeta[j] = ColumnMetadata.read(in); - } - else + } else _colmeta[j] = new ColumnMetadata(); // must be allocated. - if(type >= 0 && _nRow > 0) // if in allocated column data then read it + if (type >= 0 && _nRow > 0) // if in allocated column data then read it _coldata[j] = ArrayFactory.read(in, _nRow); } _msize = -1; @@ -890,7 +910,7 @@ public void readExternal(ObjectInput in) throws IOException { @Override public long getInMemorySize() { // reuse previously computed size - if(_msize > 0) + if (_msize > 0) return _msize; // frame block header @@ -906,8 +926,8 @@ public long getInMemorySize() { // meta data array (overhead and entries) size += MemoryEstimates.objectArrayCost(clen); - if( _colmeta != null ) - for(ColumnMetadata mtd : _colmeta) + if (_colmeta != null) + for (ColumnMetadata mtd : _colmeta) size += mtd == null ? 8 : mtd.getInMemorySize(); // data array @@ -921,35 +941,32 @@ private double arraysSizeInMemory() { final int clen = getNumColumns(); final int rlen = getNumRows(); double size = 0; - if(_coldata == null) // not allocated estimate if allocated - for(int j = 0; j < clen; j++) + if (_coldata == null) // not allocated estimate if allocated + for (int j = 0; j < clen; j++) size += ArrayFactory.getInMemorySize(_schema[j], rlen, true); else {// allocated - if((rlen > 1000 || clen > 10 )&& ConfigurationManager.isParallelIOEnabled()) { + if ((rlen > 1000 || clen > 10) && ConfigurationManager.isParallelIOEnabled()) { final ExecutorService pool = CommonThreadPool.get(); try { List> f = new ArrayList<>(clen); - for(int i = 0; i < clen; i++) { + for (int i = 0; i < clen; i++) { final int j = i; f.add(pool.submit(() -> _coldata[j].getInMemorySize())); } - for(Future e : f) { + for (Future e : f) { size += e.get(); } - } - catch(InterruptedException | ExecutionException e) { + } catch (InterruptedException | ExecutionException e) { LOG.error(e); size = 0; - for(Array aa : _coldata) + for (Array aa : _coldata) size += aa.getInMemorySize(); - } - finally { + } finally { pool.shutdown(); } - } - else { - for(Array aa : _coldata) + } else { + for (Array aa : _coldata) size += aa.getInMemorySize(); } } @@ -964,13 +981,13 @@ public long getExactSerializedSize() { size += 1 * getNumColumns(); // column schema // column sizes final boolean isDefaultMeta = isColNamesDefault() && isColumnMetadataDefault(); - for(int j = 0; j < getNumColumns(); j++) { + for (int j = 0; j < getNumColumns(); j++) { final byte type = getTypeForIO(j); - if(!isDefaultMeta) { + if (!isDefaultMeta) { size += IOUtilFunctions.getUTFSize(getColumnName(j)); size += _colmeta[j].getExactSerializedSize(); } - if(type > 0) + if (type > 0) size += _coldata[j].getExactSerializedSize(); } return size; @@ -985,9 +1002,9 @@ public boolean isShallowSerialize() { public boolean isShallowSerialize(boolean inclConvert) { // shallow serialize if non-string schema because a frame block // is always dense but strings have large array overhead per cell - if( _schema != null ) - for(int j = 0; j < _schema.length; j++) - if(!_coldata[j].isShallowSerialize()) + if (_schema != null) + for (int j = 0; j < _schema.length; j++) + if (!_coldata[j].isShallowSerialize()) return false; return true; } @@ -1013,56 +1030,52 @@ public void compactEmptyBlock() { * @return a boolean frameBlock */ public FrameBlock binaryOperations(BinaryOperator bop, FrameBlock that, FrameBlock out) { - if(getNumColumns() != that.getNumColumns() && getNumRows() != that.getNumColumns()) + if (getNumColumns() != that.getNumColumns() && getNumRows() != that.getNumColumns()) throw new DMLRuntimeException("Frame dimension mismatch " + getNumRows() + " * " + getNumColumns() + " != " - + that.getNumRows() + " * " + that.getNumColumns()); + + that.getNumRows() + " * " + that.getNumColumns()); String[][] outputData = new String[getNumRows()][getNumColumns()]; // compare output value, incl implicit type promotion if necessary - if(bop.fn instanceof ValueComparisonFunction) { + if (bop.fn instanceof ValueComparisonFunction) { ValueComparisonFunction vcomp = (ValueComparisonFunction) bop.fn; out = executeValueComparisons(this, that, vcomp, outputData); - } - else + } else throw new DMLRuntimeException("Unsupported binary operation on frames (only comparisons supported)"); return out; } private FrameBlock executeValueComparisons(FrameBlock frameBlock, FrameBlock that, ValueComparisonFunction vcomp, - String[][] outputData) { - for(int i = 0; i < getNumColumns(); i++) { - if(getSchema()[i] == ValueType.STRING || that.getSchema()[i] == ValueType.STRING) { - for(int j = 0; j < getNumRows(); j++) { - if(checkAndSetEmpty(frameBlock, that, outputData, j, i)) + String[][] outputData) { + for (int i = 0; i < getNumColumns(); i++) { + if (getSchema()[i] == ValueType.STRING || that.getSchema()[i] == ValueType.STRING) { + for (int j = 0; j < getNumRows(); j++) { + if (checkAndSetEmpty(frameBlock, that, outputData, j, i)) continue; String v1 = UtilFunctions.objectToString(get(j, i)); String v2 = UtilFunctions.objectToString(that.get(j, i)); outputData[j][i] = String.valueOf(vcomp.compare(v1, v2)); } - } - else if(getSchema()[i] == ValueType.FP64 || that.getSchema()[i] == ValueType.FP64 || - getSchema()[i] == ValueType.FP32 || that.getSchema()[i] == ValueType.FP32) { - for(int j = 0; j < getNumRows(); j++) { - if(checkAndSetEmpty(frameBlock, that, outputData, j, i)) + } else if (getSchema()[i] == ValueType.FP64 || that.getSchema()[i] == ValueType.FP64 || + getSchema()[i] == ValueType.FP32 || that.getSchema()[i] == ValueType.FP32) { + for (int j = 0; j < getNumRows(); j++) { + if (checkAndSetEmpty(frameBlock, that, outputData, j, i)) continue; ScalarObject so1 = new DoubleObject(Double.parseDouble(get(j, i).toString())); ScalarObject so2 = new DoubleObject(Double.parseDouble(that.get(j, i).toString())); outputData[j][i] = String.valueOf(vcomp.compare(so1.getDoubleValue(), so2.getDoubleValue())); } - } - else if(getSchema()[i] == ValueType.INT64 || that.getSchema()[i] == ValueType.INT64 || - getSchema()[i] == ValueType.INT32 || that.getSchema()[i] == ValueType.INT32) { - for(int j = 0; j < this.getNumRows(); j++) { - if(checkAndSetEmpty(frameBlock, that, outputData, j, i)) + } else if (getSchema()[i] == ValueType.INT64 || that.getSchema()[i] == ValueType.INT64 || + getSchema()[i] == ValueType.INT32 || that.getSchema()[i] == ValueType.INT32) { + for (int j = 0; j < this.getNumRows(); j++) { + if (checkAndSetEmpty(frameBlock, that, outputData, j, i)) continue; ScalarObject so1 = new IntObject(Integer.parseInt(get(j, i).toString())); ScalarObject so2 = new IntObject(Integer.parseInt(that.get(j, i).toString())); outputData[j][i] = String.valueOf(vcomp.compare(so1.getLongValue(), so2.getLongValue())); } - } - else { - for(int j = 0; j < getNumRows(); j++) { - if(checkAndSetEmpty(frameBlock, that, outputData, j, i)) + } else { + for (int j = 0; j < getNumRows(); j++) { + if (checkAndSetEmpty(frameBlock, that, outputData, j, i)) continue; ScalarObject so1 = new BooleanObject(Boolean.parseBoolean(get(j, i).toString())); ScalarObject so2 = new BooleanObject(Boolean.parseBoolean(that.get(j, i).toString())); @@ -1074,7 +1087,7 @@ else if(getSchema()[i] == ValueType.INT64 || that.getSchema()[i] == ValueType.IN } private static boolean checkAndSetEmpty(FrameBlock fb1, FrameBlock fb2, String[][] out, int r, int c) { - if(fb1.get(r, c) == null || fb2.get(r, c) == null) { + if (fb1.get(r, c) == null || fb2.get(r, c) == null) { out[r][c] = (fb1.get(r, c) == null && fb2.get(r, c) == null) ? "true" : "false"; return true; } @@ -1083,27 +1096,27 @@ private static boolean checkAndSetEmpty(FrameBlock fb1, FrameBlock fb2, String[] public FrameBlock leftIndexingOperations(FrameBlock rhsFrame, IndexRange ixrange, FrameBlock ret) { return leftIndexingOperations(rhsFrame, (int) ixrange.rowStart, (int) ixrange.rowEnd, (int) ixrange.colStart, - (int) ixrange.colEnd, ret); + (int) ixrange.colEnd, ret); } public FrameBlock leftIndexingOperations(FrameBlock rhsFrame, int rl, int ru, int cl, int cu, FrameBlock ret) { // check the validity of bounds - if(rl < 0 || rl >= getNumRows() || ru < rl || ru >= getNumRows() || cl < 0 || cu >= getNumColumns() || - cu < cl || cu >= getNumColumns()) { + if (rl < 0 || rl >= getNumRows() || ru < rl || ru >= getNumRows() || cl < 0 || cu >= getNumColumns() || + cu < cl || cu >= getNumColumns()) { throw new DMLRuntimeException( - "Invalid values for frame indexing: [" + (rl + 1) + ":" + (ru + 1) + "," + (cl + 1) + ":" + (cu + 1) - + "] " + "must be within frame dimensions [" + getNumRows() + "," + getNumColumns() + "]."); + "Invalid values for frame indexing: [" + (rl + 1) + ":" + (ru + 1) + "," + (cl + 1) + ":" + (cu + 1) + + "] " + "must be within frame dimensions [" + getNumRows() + "," + getNumColumns() + "]."); } - if((ru - rl + 1) < rhsFrame.getNumRows() || (cu - cl + 1) < rhsFrame.getNumColumns()) { + if ((ru - rl + 1) < rhsFrame.getNumRows() || (cu - cl + 1) < rhsFrame.getNumColumns()) { throw new DMLRuntimeException( - "Invalid values for frame indexing: " + "dimensions of the source frame [" + rhsFrame.getNumRows() + "x" - + rhsFrame.getNumColumns() + "] " + "do not match the shape of the frame specified by indices [" - + (rl + 1) + ":" + (ru + 1) + ", " + (cl + 1) + ":" + (cu + 1) + "]."); + "Invalid values for frame indexing: " + "dimensions of the source frame [" + rhsFrame.getNumRows() + "x" + + rhsFrame.getNumColumns() + "] " + "do not match the shape of the frame specified by indices [" + + (rl + 1) + ":" + (ru + 1) + ", " + (cl + 1) + ":" + (cu + 1) + "]."); } // allocate output frame (incl deep copy schema) - if(ret == null) + if (ret == null) ret = new FrameBlock(); ret._schema = _schema.clone(); @@ -1113,15 +1126,15 @@ public FrameBlock leftIndexingOperations(FrameBlock rhsFrame, int rl, int ru, in ret._nRow = _nRow; // copy data to output and partial overwrite w/ rhs - for(int j = 0; j < getNumColumns(); j++) { + for (int j = 0; j < getNumColumns(); j++) { Array tmp = _coldata[j].clone(); - if(j >= cl && j <= cu) { + if (j >= cl && j <= cu) { // fast-path for homogeneous column schemas - if(_schema[j] == rhsFrame._schema[j - cl]) + if (_schema[j] == rhsFrame._schema[j - cl]) tmp.set(rl, ru, rhsFrame._coldata[j - cl]); - // general-path for heterogeneous column schemas + // general-path for heterogeneous column schemas else { - for(int i = rl; i <= ru; i++) + for (int i = rl; i <= ru; i++) tmp.set(i, UtilFunctions.objectToObject(_schema[j], rhsFrame._coldata[j - cl].get(i - rl))); } } @@ -1165,7 +1178,7 @@ public final FrameBlock slice(int rl, int ru, int cl, int cu, boolean deep) { public FrameBlock slice(int rl, int ru, int cl, int cu, boolean deep, FrameBlock ret) { validateSliceArgument(rl, ru, cl, cu); // allocate output frame - if(ret == null) + if (ret == null) ret = new FrameBlock(); // copy output schema and colnames @@ -1177,27 +1190,27 @@ public FrameBlock slice(int rl, int ru, int cl, int cu, boolean deep, FrameBlock ret._colmeta = new ColumnMetadata[numCols]; // names - for(int j = cl; j <= cu; j++) { + for (int j = cl; j <= cu; j++) { ret._schema[j - cl] = _schema[j]; ret._colmeta[j - cl] = _colmeta[j]; - if(!isDefNames) + if (!isDefNames) ret._colnames[j - cl] = getColumnName(j); } - if(ret._coldata == null) + if (ret._coldata == null) ret._coldata = new Array[numCols]; // fast-path: shallow copy column indexing - if(ret.getNumRows() == getNumRows() && !deep) { + if (ret.getNumRows() == getNumRows() && !deep) { // this shallow copy does not only avoid an array copy, but // also allows for bi-directional reuses of recodemaps - for(int j = cl; j <= cu; j++) + for (int j = cl; j <= cu; j++) ret._coldata[j - cl] = _coldata[j]; } // copy output data else { - for(int j = cl; j <= cu; j++) { - if(ret._coldata[j - cl] == null) + for (int j = cl; j <= cu; j++) { + if (ret._coldata[j - cl] == null) ret._coldata[j - cl] = _coldata[j].slice(rl, ru + 1); else ret._coldata[j - cl].set(0, ru - rl, _coldata[j], rl); @@ -1208,24 +1221,24 @@ public FrameBlock slice(int rl, int ru, int cl, int cu, boolean deep, FrameBlock } protected void validateSliceArgument(int rl, int ru, int cl, int cu) { - if(rl < 0 || rl >= getNumRows() || ru < rl || ru >= getNumRows() || cl < 0 || cu >= getNumColumns() || - cu < cl || cu >= getNumColumns()) { + if (rl < 0 || rl >= getNumRows() || ru < rl || ru >= getNumRows() || cl < 0 || cu >= getNumColumns() || + cu < cl || cu >= getNumColumns()) { throw new DMLRuntimeException( - "Invalid values for frame indexing: [" + (rl + 1) + ":" + (ru + 1) + "," + (cl + 1) + ":" + (cu + 1) - + "] " + "must be within frame dimensions [" + getNumRows() + "," + getNumColumns() + "]"); + "Invalid values for frame indexing: [" + (rl + 1) + ":" + (ru + 1) + "," + (cl + 1) + ":" + (cu + 1) + + "] " + "must be within frame dimensions [" + getNumRows() + "," + getNumColumns() + "]"); } } public void slice(ArrayList> outList, IndexRange range, int rowCut) { - if(getNumRows() > 0) { - if(outList.size() > 1) + if (getNumRows() > 0) { + if (outList.size() > 1) throw new NotImplementedException("Not implemented slice of more than 1 block out"); int r = (int) range.rowStart; final FrameBlock out = outList.get(0).getValue(); - if(range.rowStart < rowCut) + if (range.rowStart < rowCut) slice(r, (int) Math.min(rowCut, range.rowEnd + 1), (int) range.colStart, (int) range.colEnd, out); - if(range.rowEnd >= rowCut) + if (range.rowEnd >= rowCut) slice(r, (int) range.rowEnd, (int) range.colStart, (int) range.colEnd, out); } @@ -1254,39 +1267,39 @@ public void copy(FrameBlock src) { int nCol = src.getNumColumns(); _nRow = src.getNumRows(); _schema = Arrays.copyOf(src._schema, nCol); - if(src._colnames != null) + if (src._colnames != null) _colnames = Arrays.copyOf(src._colnames, nCol); - if(!src.isColumnMetadataDefault()) + if (!src.isColumnMetadataDefault()) _colmeta = Arrays.copyOf(src._colmeta, nCol); - if(src._coldata != null) { + if (src._coldata != null) { _coldata = new Array[nCol]; - for(int i = 0; i < nCol; i++) + for (int i = 0; i < nCol; i++) _coldata[i] = src._coldata[i].clone(); } _msize = -1; } - public FrameBlock copyShallow(){ + public FrameBlock copyShallow() { FrameBlock ret = new FrameBlock(); ret._nRow = _nRow; - ret._msize = _msize; + ret._msize = _msize; final int nCol = getNumColumns(); - if(_coldata != null) + if (_coldata != null) ret._coldata = Arrays.copyOf(_coldata, nCol); - if(_colnames != null) + if (_colnames != null) ret._colnames = Arrays.copyOf(_colnames, nCol); - if(_colmeta != null) + if (_colmeta != null) ret._colmeta = Arrays.copyOf(_colmeta, nCol); - if(_schema != null) + if (_schema != null) ret._schema = Arrays.copyOf(_schema, nCol); return ret; } /** * Copy src matrix into the index range of the existing current matrix. - * + *

* This is used to copy smaller blocks into a larger block, for instance in binary reading. - * + * * @param rl row start * @param ru row end inclusive * @param cl col start @@ -1295,25 +1308,25 @@ public FrameBlock copyShallow(){ */ public void copy(int rl, int ru, int cl, int cu, FrameBlock src) { // If full copy, fall back to default copy - if(rl == 0 && cl == 0 && ru + 1 == this.getNumRows() && cu + 1 == this.getNumColumns()) { + if (rl == 0 && cl == 0 && ru + 1 == this.getNumRows() && cu + 1 == this.getNumColumns()) { copy(src); return; } ensureAllocateMeta(); - if(_coldata == null) // allocate column data. + if (_coldata == null) // allocate column data. _coldata = new Array[_schema.length]; - synchronized(this) { // make sync locks + synchronized (this) { // make sync locks // TODO remove sync locks on array types where they are not needed. - if(_columnLocks == null) { + if (_columnLocks == null) { Object[] locks = new Object[_schema.length]; - for(int i = 0; i < locks.length; i++) + for (int i = 0; i < locks.length; i++) locks[i] = new Object(); _columnLocks = new SoftReference<>(locks); } } Object[] locks = _columnLocks.get(); - for(int j = cl; j <= cu; j++) { // for each column - synchronized(locks[j]) { // synchronize on the column. + for (int j = cl; j <= cu; j++) { // for each column + synchronized (locks[j]) { // synchronize on the column. _coldata[j] = ArrayFactory.set(_coldata[j], src._coldata[j - cl], rl, ru, _nRow); } } @@ -1338,25 +1351,25 @@ public FrameBlock merge(FrameBlock that, boolean appendOnly) { public FrameBlock merge(FrameBlock that) { // check for empty input source (nothing to merge) - if(that == null || that.getNumRows() == 0) + if (that == null || that.getNumRows() == 0) return this; // check dimensions (before potentially copy to prevent implicit dimension change) - if(getNumRows() != that.getNumRows() || getNumColumns() != that.getNumColumns()) + if (getNumRows() != that.getNumRows() || getNumColumns() != that.getNumColumns()) throw new DMLRuntimeException("Dimension mismatch on merge disjoint (target=" + getNumRows() + "x" - + getNumColumns() + ", source=" + that.getNumRows() + "x" + that.getNumColumns() + ")"); + + getNumColumns() + ", source=" + that.getNumRows() + "x" + that.getNumColumns() + ")"); // meta data copy if necessary - for(int j = 0; j < getNumColumns(); j++) - if(!that.isColumnMetadataDefault(j)) { + for (int j = 0; j < getNumColumns(); j++) + if (!that.isColumnMetadataDefault(j)) { _colmeta[j].setNumDistinct(that._colmeta[j].getNumDistinct()); _colmeta[j].setMvValue(that._colmeta[j].getMvValue()); } // core frame block merge through cell copy // with column-wide access pattern - for(int j = 0; j < getNumColumns(); j++) { - if(_coldata[j].getValueType().equals(that._coldata[j].getValueType())) + for (int j = 0; j < getNumColumns(); j++) { + if (_coldata[j].getValueType().equals(that._coldata[j].getValueType())) _coldata[j].setNz(that._coldata[j]); else _coldata[j].setFromOtherTypeNz(that._coldata[j]); @@ -1377,10 +1390,10 @@ public FrameBlock merge(FrameBlock that) { * @return frame block */ public FrameBlock zeroOutOperations(FrameBlock result, IndexRange range, boolean complementary, int iRowStartSrc, - int iRowStartDest, int blen, int iMaxRowsToCopy) { + int iRowStartDest, int blen, int iMaxRowsToCopy) { int clen = getNumColumns(); - if(result == null) + if (result == null) result = new FrameBlock(getSchema()); else { result.reset(0, true); @@ -1388,28 +1401,27 @@ public FrameBlock zeroOutOperations(FrameBlock result, IndexRange range, boolean } result.ensureAllocatedColumns(blen); - if(complementary) { - for(int r = (int) range.rowStart; r <= range.rowEnd && r + iRowStartDest < blen; r++) { - for(int c = (int) range.colStart; c <= range.colEnd; c++) + if (complementary) { + for (int r = (int) range.rowStart; r <= range.rowEnd && r + iRowStartDest < blen; r++) { + for (int c = (int) range.colStart; c <= range.colEnd; c++) result.set(r + iRowStartDest, c, get(r + iRowStartSrc, c)); } - } - else { + } else { int r = iRowStartDest; - for(; r < (int) range.rowStart && r - iRowStartDest < iMaxRowsToCopy; r++) - for(int c = 0; c < clen; c++/* , offset++ */) + for (; r < (int) range.rowStart && r - iRowStartDest < iMaxRowsToCopy; r++) + for (int c = 0; c < clen; c++/* , offset++ */) result.set(r, c, get(r + iRowStartSrc - iRowStartDest, c)); - for(; r <= (int) range.rowEnd && r - iRowStartDest < iMaxRowsToCopy; r++) { - for(int c = 0; c < (int) range.colStart; c++) + for (; r <= (int) range.rowEnd && r - iRowStartDest < iMaxRowsToCopy; r++) { + for (int c = 0; c < (int) range.colStart; c++) result.set(r, c, get(r + iRowStartSrc - iRowStartDest, c)); - for(int c = (int) range.colEnd + 1; c < clen; c++) + for (int c = (int) range.colEnd + 1; c < clen; c++) result.set(r, c, get(r + iRowStartSrc - iRowStartDest, c)); } - for(; r - iRowStartDest < iMaxRowsToCopy; r++) - for(int c = 0; c < clen; c++) + for (; r - iRowStartDest < iMaxRowsToCopy; r++) + for (int c = 0; c < clen; c++) result.set(r, c, get(r + iRowStartSrc - iRowStartDest, c)); } @@ -1440,43 +1452,43 @@ public final FrameBlock applySchema(FrameBlock schema, int k) { /** * Drop the cell value which does not confirms to the data type of its column - * + * * @param schema of the frame * @return original frame where invalid values are replaced with null */ public FrameBlock dropInvalidType(FrameBlock schema) { // sanity checks - if(this.getNumColumns() != schema.getNumColumns()) + if (this.getNumColumns() != schema.getNumColumns()) throw new DMLException("mismatch in number of columns in frame and its schema " + this.getNumColumns() - + " != " + schema.getNumColumns()); + + " != " + schema.getNumColumns()); // extract the schema in String array String[] schemaString = IteratorFactory.getStringRowIterator(schema).next(); - for(int i = 0; i < this.getNumColumns(); i++) { + for (int i = 0; i < this.getNumColumns(); i++) { Array obj = this.getColumn(i); String schemaCol = schemaString[i]; String type; - if(schemaCol.contains("FP")) + if (schemaCol.contains("FP")) type = "FP"; - else if(schemaCol.contains("INT")) + else if (schemaCol.contains("INT")) type = "INT"; - else if(schemaCol.contains("STRING")) + else if (schemaCol.contains("STRING")) // In case of String columns, don't do any verification or replacements. continue; else type = schemaCol; - for(int j = 0; j < this.getNumRows(); j++) { - if(obj.get(j) == null) + for (int j = 0; j < this.getNumRows(); j++) { + if (obj.get(j) == null) continue; String dataValue = obj.get(j).toString().trim().replace("\"", "").toLowerCase(); ValueType dataType = FrameUtil.isType(dataValue); - if(!dataType.toString().contains(type) && !(dataType == ValueType.BOOLEAN && type.equals("INT")) && - !(dataType == ValueType.BOOLEAN && type.equals("FP"))) { + if (!dataType.toString().contains(type) && !(dataType == ValueType.BOOLEAN && type.equals("INT")) && + !(dataType == ValueType.BOOLEAN && type.equals("FP"))) { LOG.warn("Datatype detected: " + dataType + " where expected: " + schemaString[i] + " col: " - + (i + 1) + ", row:" + (j + 1)); + + (i + 1) + ", row:" + (j + 1)); this.set(j, i, null); } @@ -1495,20 +1507,20 @@ else if(schemaCol.contains("STRING")) */ public FrameBlock invalidByLength(MatrixBlock feaLen) { // sanity checks - if(this.getNumColumns() != feaLen.getNumColumns()) + if (this.getNumColumns() != feaLen.getNumColumns()) throw new DMLException("mismatch in number of columns in frame and corresponding feature-length vector"); FrameBlock outBlock = new FrameBlock(this); - for(int i = 0; i < this.getNumColumns(); i++) { - if(feaLen.get(0, i) == -1) + for (int i = 0; i < this.getNumColumns(); i++) { + if (feaLen.get(0, i) == -1) continue; int validLength = (int) feaLen.get(0, i); Array obj = this.getColumn(i); - for(int j = 0; j < obj.size(); j++) { - if(obj.get(j) == null) + for (int j = 0; j < obj.size(); j++) { + if (obj.get(j) == null) continue; String dataValue = obj.get(j).toString(); - if(dataValue.length() > validLength) + if (dataValue.length() > validLength) outBlock.set(j, i, null); } } @@ -1517,40 +1529,39 @@ public FrameBlock invalidByLength(MatrixBlock feaLen) { } public void mapInplace(Function fun) { - for(int j = 0; j < getNumColumns(); j++) - for(int i = 0; i < getNumRows(); i++) { + for (int j = 0; j < getNumColumns(); j++) + for (int i = 0; i < getNumRows(); i++) { Object tmp = get(i, j); set(i, j, (tmp == null) ? tmp : UtilFunctions.objectToObject(_schema[j], fun.apply(tmp.toString()))); } } public FrameBlock map(String lambdaExpr, long margin) { - if(!lambdaExpr.contains("->")) { + if (!lambdaExpr.contains("->")) { String args = lambdaExpr.substring(lambdaExpr.indexOf('(') + 1, lambdaExpr.indexOf(')')); - if(args.contains(",")) { + if (args.contains(",")) { String[] arguments = args.split(","); return DMVUtils.syntacticalPatternDiscovery(this, Double.parseDouble(arguments[0]), arguments[1]); - } - else if(args.contains(";")) { + } else if (args.contains(";")) { String[] arguments = args.split(";"); return EMAUtils.exponentialMovingAverageImputation(this, Integer.parseInt(arguments[0]), arguments[1], - Integer.parseInt(arguments[2]), Double.parseDouble(arguments[3]), Double.parseDouble(arguments[4]), - Double.parseDouble(arguments[5])); + Integer.parseInt(arguments[2]), Double.parseDouble(arguments[3]), Double.parseDouble(arguments[4]), + Double.parseDouble(arguments[5])); } } - if(lambdaExpr.contains("jaccardSim")) + if (lambdaExpr.contains("jaccardSim")) return mapDist(getCompiledFunction(lambdaExpr, margin)); return map(getCompiledFunction(lambdaExpr, margin), margin); } public FrameBlock frameRowReplication(FrameBlock rowToreplicate) { FrameBlock out = new FrameBlock(this); - if(this.getNumColumns() != rowToreplicate.getNumColumns()) + if (this.getNumColumns() != rowToreplicate.getNumColumns()) throw new DMLRuntimeException("Mismatch number of columns"); - if(rowToreplicate.getNumRows() > 1) + if (rowToreplicate.getNumRows() > 1) throw new DMLRuntimeException("only supported single rows frames to replicate"); - for(int i = 0; i < this.getNumRows(); i++) - for(int j = 0; j < this.getNumColumns(); j++) + for (int i = 0; i < this.getNumRows(); i++) + for (int j = 0; j < this.getNumColumns(); j++) out.set(i, j, rowToreplicate.get(0, j)); return out; } @@ -1562,42 +1573,42 @@ public FrameBlock valueSwap(FrameBlock schema) { double minSimScore = 0; int bestIdx = 0; // remove the precision info - for(int i = 0; i < schemaString.length; i++) + for (int i = 0; i < schemaString.length; i++) schemaString[i] = schemaString[i].replaceAll("\\d", ""); double[] minColLength = new double[this.getNumColumns()]; double[] maxColLength = new double[this.getNumColumns()]; - for(int k = 0; k < this.getNumColumns(); k++) { + for (int k = 0; k < this.getNumColumns(); k++) { Pair minMax = _coldata[k].getMinMaxLength(); maxColLength[k] = minMax.getKey(); minColLength[k] = minMax.getValue(); } ArrayList probColList = new ArrayList(); - for(int i = 0; i < this.getNumColumns(); i++) { - for(int j = 0; j < this.getNumRows(); j++) { - if(this.get(j, i) == null) + for (int i = 0; i < this.getNumColumns(); i++) { + for (int j = 0; j < this.getNumRows(); j++) { + if (this.get(j, i) == null) continue; String dataValue = this.get(j, i).toString().trim().replace("\"", "").toLowerCase(); ValueType dataType = FrameUtil.isType(dataValue); String type = dataType.toString().replaceAll("\\d", ""); // get the avergae column length - if(!dataType.toString().contains(schemaString[i]) && - !(dataType == ValueType.BOOLEAN && schemaString[i].equals("INT")) && - !(dataType == ValueType.BOOLEAN && schemaString[i].equals("FP")) && - !(dataType.toString().contains("INT") && schemaString[i].equals("FP"))) { + if (!dataType.toString().contains(schemaString[i]) && + !(dataType == ValueType.BOOLEAN && schemaString[i].equals("INT")) && + !(dataType == ValueType.BOOLEAN && schemaString[i].equals("FP")) && + !(dataType.toString().contains("INT") && schemaString[i].equals("FP"))) { LOG.warn("conflict " + dataType + " " + schemaString[i] + " " + dataValue); // check the other column with satisfy the data type of this value - for(int w = 0; w < schemaString.length; w++) { - if(schemaString[w].equals(type) && dataValue.length() > minColLength[w] && - dataValue.length() < maxColLength[w] && (w != i)) { + for (int w = 0; w < schemaString.length; w++) { + if (schemaString[w].equals(type) && dataValue.length() > minColLength[w] && + dataValue.length() < maxColLength[w] && (w != i)) { Object item = this.get(j, w); String dataValueProb = (item != null) ? item.toString().trim().replace("\"", "") - .toLowerCase() : "0"; + .toLowerCase() : "0"; ValueType dataTypeProb = FrameUtil.isType(dataValueProb); - if(!dataTypeProb.toString().equals(schemaString[w])) { + if (!dataTypeProb.toString().equals(schemaString[w])) { bestIdx = w; break; } @@ -1606,25 +1617,24 @@ public FrameBlock valueSwap(FrameBlock schema) { } // if we have more than one column that is the probable match for this value then find the most // appropriate one by using the similarity score - if(probColList.size() > 1) { - for(int w : probColList) { + if (probColList.size() > 1) { + for (int w : probColList) { int randomIndex = ThreadLocalRandom.current().nextInt(0, getNumRows() - 1); Object value = this.get(randomIndex, w); - if(value != null) { + if (value != null) { dataValue2 = value.toString(); } // compute distance between sample and invalid value double simScore = 0; - if(!(dataValue == null) && !(dataValue2 == null)) + if (!(dataValue == null) && !(dataValue2 == null)) simScore = StringUtils.getLevenshteinDistance(dataValue, dataValue2); - if(simScore < minSimScore) { + if (simScore < minSimScore) { minSimScore = simScore; bestIdx = w; } } - } - else if(probColList.size() > 0) { + } else if (probColList.size() > 0) { bestIdx = probColList.get(0); } String tmp = dataValue; @@ -1640,33 +1650,31 @@ public FrameBlock map(FrameMapFunction lambdaExpr, long margin) { // Prepare temporary output array String[][] output = new String[getNumRows()][getNumColumns()]; - if(margin == 1) { + if (margin == 1) { // Execute map function on rows - for(int i = 0; i < getNumRows(); i++) { + for (int i = 0; i < getNumRows(); i++) { String[] row = new String[getNumColumns()]; - for(int j = 0; j < getNumColumns(); j++) { + for (int j = 0; j < getNumColumns(); j++) { Array input = getColumn(j); row[j] = String.valueOf(input.get(i)); } output[i] = lambdaExpr.apply(row); } - } - else if(margin == 2) { + } else if (margin == 2) { // Execute map function on columns - for(int j = 0; j < getNumColumns(); j++) { + for (int j = 0; j < getNumColumns(); j++) { // since more rows can be allocated, mutable array String[] actualColumn = Arrays.copyOfRange((String[]) getColumnData(j), 0, getNumRows()); String[] outColumn = lambdaExpr.apply(actualColumn); - for(int i = 0; i < getNumRows(); i++) + for (int i = 0; i < getNumRows(); i++) output[i][j] = outColumn[i]; } - } - else { + } else { // Execute map function on all cells - for(int j = 0; j < getNumColumns(); j++) { + for (int j = 0; j < getNumColumns(); j++) { Array input = getColumn(j); - for(int i = 0; i < input.size(); i++) - if(input.get(i) != null) + for (int i = 0; i < input.size(); i++) + if (input.get(i) != null) output[i][j] = lambdaExpr.apply(String.valueOf(input.get(i))); } } @@ -1675,12 +1683,12 @@ else if(margin == 2) { public FrameBlock mapDist(FrameMapFunction lambdaExpr) { String[][] output = new String[getNumRows()][getNumRows()]; - for(String[] row : output) + for (String[] row : output) Arrays.fill(row, "0.0"); Array input = getColumn(0); - for(int j = 0; j < input.size() - 1; j++) { - for(int i = j + 1; i < input.size(); i++) - if(input.get(i) != null && input.get(j) != null) { + for (int j = 0; j < input.size() - 1; j++) { + for (int i = j + 1; i < input.size(); i++) + if (input.get(i) != null && input.get(j) != null) { output[j][i] = lambdaExpr.apply(String.valueOf(input.get(j)), String.valueOf(input.get(i))); } } @@ -1691,7 +1699,7 @@ public static FrameMapFunction getCompiledFunction(String lambdaExpr, long margi String cname = "StringProcessing" + CLASS_ID.getNextID(); StringBuilder sb = new StringBuilder(); String[] parts = lambdaExpr.split("->"); - if(parts.length != 2) + if (parts.length != 2) throw new DMLRuntimeException("Unsupported lambda expression: " + lambdaExpr); String[] varname = parts[0].replaceAll("[()]", "").split(","); String expr = parts[1].trim(); @@ -1702,28 +1710,25 @@ public static FrameMapFunction getCompiledFunction(String lambdaExpr, long margi sb.append("import org.apache.sysds.runtime.frame.data.FrameBlock.FrameMapFunction;\n"); sb.append("import java.util.Arrays;\n"); sb.append("public class " + cname + " extends FrameMapFunction {\n"); - if(margin != 0) { + if (margin != 0) { sb.append("public String[] apply(String[] " + varname[0].trim() + ") {\n"); sb.append(" return UtilFunctions.toStringArray(" + expr + "); }}\n"); - } - else { - if(varname.length == 1) { + } else { + if (varname.length == 1) { sb.append("public String apply(String " + varname[0].trim() + ") {\n"); sb.append(" return String.valueOf(" + expr + "); }}\n"); - } - else if(varname.length == 2) { + } else if (varname.length == 2) { sb.append( - "public String apply(String " + varname[0].trim() + ", String " + varname[1].trim() + ") {\n"); + "public String apply(String " + varname[0].trim() + ", String " + varname[1].trim() + ") {\n"); sb.append(" return String.valueOf(" + expr + "); }}\n"); } } // compile class, and create FrameMapFunction object try { return (FrameMapFunction) CodegenUtils.compileClass(cname, sb.toString()).getDeclaredConstructor() - .newInstance(); - } - catch(InstantiationException | IllegalAccessException | IllegalArgumentException | InvocationTargetException - | NoSuchMethodException | SecurityException e) { + .newInstance(); + } catch (InstantiationException | IllegalAccessException | IllegalArgumentException | InvocationTargetException + | NoSuchMethodException | SecurityException e) { throw new DMLRuntimeException("Failed to compile FrameMapFunction.", e); } } @@ -1750,29 +1755,29 @@ public FrameBlock replaceOperations(String pattern, String replacement) { boolean NaNp = "NaN".equals(pattern); boolean NaNr = "NaN".equals(replacement); ValueType patternType = UtilFunctions - .isBoolean(pattern) ? ValueType.BOOLEAN : (NumberUtils.isCreatable(pattern) | + .isBoolean(pattern) ? ValueType.BOOLEAN : (NumberUtils.isCreatable(pattern) | NaNp ? (UtilFunctions.isIntegerNumber(pattern) ? ValueType.INT64 : ValueType.FP64) : ValueType.STRING); ValueType replacementType = UtilFunctions.isBoolean(replacement) ? ValueType.BOOLEAN : (NumberUtils - .isCreatable(replacement) | - NaNr ? (UtilFunctions.isIntegerNumber(replacement) ? ValueType.INT64 : ValueType.FP64) : ValueType.STRING); + .isCreatable(replacement) | + NaNr ? (UtilFunctions.isIntegerNumber(replacement) ? ValueType.INT64 : ValueType.FP64) : ValueType.STRING); - if(patternType != replacementType || !ValueType.isSameTypeString(patternType, replacementType)) + if (patternType != replacementType || !ValueType.isSameTypeString(patternType, replacementType)) throw new DMLRuntimeException( - "Pattern and replacement types should be same: " + patternType + " " + replacementType); + "Pattern and replacement types should be same: " + patternType + " " + replacementType); - for(int i = 0; i < ret.getNumColumns(); i++) { + for (int i = 0; i < ret.getNumColumns(); i++) { Array colData = ret._coldata[i]; - for(int j = 0; - j < colData.size() && - (ValueType.isSameTypeString(_schema[i], patternType) || _schema[i] == ValueType.STRING); - j++) { + for (int j = 0; + j < colData.size() && + (ValueType.isSameTypeString(_schema[i], patternType) || _schema[i] == ValueType.STRING); + j++) { T patternNew = (T) UtilFunctions.stringToObject(_schema[i], pattern); T replacementNew = (T) UtilFunctions.stringToObject(_schema[i], replacement); Object ent = colData.get(j); - if(ent != null && ent.toString().equals(patternNew.toString())) + if (ent != null && ent.toString().equals(patternNew.toString())) colData.set(j, replacementNew); - else if(ent instanceof String && ent.equals(pattern)) + else if (ent instanceof String && ent.equals(pattern)) colData.set(j, replacement); } } @@ -1787,19 +1792,19 @@ public FrameBlock removeEmptyOperations(boolean rows, boolean emptyReturn, Matri public String toString() { StringBuilder sb = new StringBuilder(); sb.append("FrameBlock"); - if(_colnames != null) { + if (_colnames != null) { sb.append("\n"); sb.append(Arrays.toString(_colnames)); } - if(!isColumnMetadataDefault()) { + if (!isColumnMetadataDefault()) { sb.append("\n"); sb.append(Arrays.toString(_colmeta)); } sb.append("\n"); sb.append(Arrays.toString(_schema)); sb.append("\n"); - if(_coldata != null) { - for(int i = 0; i < _coldata.length; i++) { + if (_coldata != null) { + for (int i = 0; i < _coldata.length; i++) { sb.append(_coldata[i]); sb.append("\n"); } diff --git a/src/main/java/org/apache/sysds/runtime/instructions/InstructionUtils.java b/src/main/java/org/apache/sysds/runtime/instructions/InstructionUtils.java index da3de02419d..805976c4a42 100644 --- a/src/main/java/org/apache/sysds/runtime/instructions/InstructionUtils.java +++ b/src/main/java/org/apache/sysds/runtime/instructions/InstructionUtils.java @@ -686,6 +686,8 @@ else if( opcode.equalsIgnoreCase(Opcodes.VALUESWAP.toString())) return new BinaryOperator(Builtin.getBuiltinFnObject("valueSwap")); else if( opcode.equalsIgnoreCase(Opcodes.FREPLICATE.toString())) return new BinaryOperator(Builtin.getBuiltinFnObject("freplicate")); + else if( opcode.equalsIgnoreCase(Opcodes.SET_COLNAMES.toString())) + return new BinaryOperator(Builtin.getBuiltinFnObject("set_colnames")); throw new RuntimeException("Unknown binary opcode " + opcode); } @@ -923,6 +925,8 @@ else if ( opcode.equalsIgnoreCase(Opcodes.DROPINVALIDLENGTH.toString()) || opcod return new BinaryOperator(Builtin.getBuiltinFnObject("dropInvalidLength")); else if ( opcode.equalsIgnoreCase(Opcodes.VALUESWAP.toString()) || opcode.equalsIgnoreCase("mapValueSwap") ) return new BinaryOperator(Builtin.getBuiltinFnObject("valueSwap")); + else if (opcode.equalsIgnoreCase(Opcodes.SET_COLNAMES.toString())) + return new BinaryOperator(Builtin.getBuiltinFnObject("set_colnames")); throw new DMLRuntimeException("Unknown binary opcode " + opcode); } diff --git a/src/main/java/org/apache/sysds/runtime/instructions/cp/BinaryFrameFrameCPInstruction.java b/src/main/java/org/apache/sysds/runtime/instructions/cp/BinaryFrameFrameCPInstruction.java index e9771b2e7fe..6d62689820d 100644 --- a/src/main/java/org/apache/sysds/runtime/instructions/cp/BinaryFrameFrameCPInstruction.java +++ b/src/main/java/org/apache/sysds/runtime/instructions/cp/BinaryFrameFrameCPInstruction.java @@ -21,6 +21,7 @@ import org.apache.sysds.common.Opcodes; import org.apache.sysds.runtime.controlprogram.context.ExecutionContext; +import org.apache.sysds.runtime.DMLRuntimeException; import org.apache.sysds.runtime.frame.data.FrameBlock; import org.apache.sysds.runtime.frame.data.lib.FrameLibApplySchema; import org.apache.sysds.runtime.matrix.operators.BinaryOperator; @@ -62,6 +63,40 @@ else if(getOpcode().equals(Opcodes.APPLYSCHEMA.toString())) { final int k = ((MultiThreadedOperator)_optr).getNumThreads(); final FrameBlock out = FrameLibApplySchema.applySchema(inBlock1, inBlock2, k); ec.setFrameOutput(output.getName(), out); + } + else if(getOpcode().equals(Opcodes.SET_COLNAMES.toString())) { + + FrameBlock in = ec.getFrameInput(input1.getName()); + FrameBlock names = ec.getFrameInput(input2.getName()); + + if (names == null) + throw new DMLRuntimeException("Column names cannot be null."); + + if (names.getNumRows() != 1) + throw new DMLRuntimeException( + "Column names must be provided as a 1 x n frame."); + + if (names.getNumColumns() != in.getNumColumns()) + throw new DMLRuntimeException( + "Expected " + in.getNumColumns() + + " column names but got " + names.getNumColumns()); + + String[] colNames = new String[(int) names.getNumColumns()]; + for(int i = 0; i < colNames.length; i++){ + colNames[i] = names.get(0, i).toString(); + } + + FrameBlock out = new FrameBlock(in); + + out.setColumnNames(colNames); + + ec.setFrameOutput(output.getName(), out); + + ec.releaseFrameInput(input1.getName()); + + ec.releaseFrameInput(input2.getName()); + + } else { // Execute binary operations diff --git a/src/main/java/org/apache/sysds/runtime/instructions/cp/UnaryFrameCPInstruction.java b/src/main/java/org/apache/sysds/runtime/instructions/cp/UnaryFrameCPInstruction.java index 107cab79d79..21d62c019db 100644 --- a/src/main/java/org/apache/sysds/runtime/instructions/cp/UnaryFrameCPInstruction.java +++ b/src/main/java/org/apache/sysds/runtime/instructions/cp/UnaryFrameCPInstruction.java @@ -52,6 +52,7 @@ else if(getOpcode().equals(Opcodes.COLNAMES.toString())) { ec.releaseFrameInput(input1.getName()); ec.setFrameOutput(output.getName(), retBlock); } + else throw new DMLScriptException("Opcode '" + getOpcode() + "' is not a valid UnaryFrameCPInstruction"); } diff --git a/src/main/java/org/apache/sysds/runtime/instructions/cp/VariableCPInstruction.java b/src/main/java/org/apache/sysds/runtime/instructions/cp/VariableCPInstruction.java index 0f707b74412..c18e46e0813 100644 --- a/src/main/java/org/apache/sysds/runtime/instructions/cp/VariableCPInstruction.java +++ b/src/main/java/org/apache/sysds/runtime/instructions/cp/VariableCPInstruction.java @@ -48,15 +48,7 @@ import org.apache.sysds.runtime.instructions.Instruction; import org.apache.sysds.runtime.instructions.InstructionUtils; import org.apache.sysds.runtime.instructions.ooc.TeeOOCInstruction; -import org.apache.sysds.runtime.io.FileFormatProperties; -import org.apache.sysds.runtime.io.FileFormatPropertiesCSV; -import org.apache.sysds.runtime.io.FileFormatPropertiesHDF5; -import org.apache.sysds.runtime.io.FileFormatPropertiesLIBSVM; -import org.apache.sysds.runtime.io.ListReader; -import org.apache.sysds.runtime.io.ListWriter; -import org.apache.sysds.runtime.io.WriterHDF5; -import org.apache.sysds.runtime.io.WriterMatrixMarket; -import org.apache.sysds.runtime.io.WriterTextCSV; +import org.apache.sysds.runtime.io.*; import org.apache.sysds.runtime.lineage.LineageItem; import org.apache.sysds.runtime.lineage.LineageItemUtils; import org.apache.sysds.runtime.lineage.LineageTraceable; @@ -91,25 +83,10 @@ public class VariableCPInstruction extends CPInstruction implements LineageTraceable { public enum VariableOperationCode { - CreateVariable, - AssignVariable, - CopyVariable, - MoveVariable, - RemoveVariable, - RemoveVariableAndFile, - CastAsScalarVariable, - CastAsMatrixVariable, - CastAsFrameVariable, - CastAsListVariable, - CastAsDoubleVariable, - CastAsIntegerVariable, - CastAsBooleanVariable, - Write, - Read, - SetFileName; + CreateVariable, AssignVariable, CopyVariable, MoveVariable, RemoveVariable, RemoveVariableAndFile, CastAsScalarVariable, CastAsMatrixVariable, CastAsFrameVariable, CastAsListVariable, CastAsDoubleVariable, CastAsIntegerVariable, CastAsBooleanVariable, Write, Read, SetFileName; public boolean isCast() { - switch(this) { + switch (this) { case CastAsScalarVariable: case CastAsMatrixVariable: case CastAsFrameVariable: @@ -125,7 +102,7 @@ public boolean isCast() { } private static final IDSequence _uniqueVarID = new IDSequence(true); - private static final int CREATEVAR_FILE_NAME_VAR_POS=3; + private static final int CREATEVAR_FILE_NAME_VAR_POS = 3; private final VariableOperationCode opcode; private final List inputs; @@ -143,8 +120,7 @@ public boolean isCast() { // CSV and LIBSVM related members (used only in createvar instructions) private final FileFormatProperties _formatProperties; - private VariableCPInstruction(VariableOperationCode op, CPOperand in1, CPOperand in2, CPOperand in3, CPOperand out, - MetaData meta, FileFormatProperties fprops, String schema, UpdateType utype, String sopcode, String istr, int k) { + private VariableCPInstruction(VariableOperationCode op, CPOperand in1, CPOperand in2, CPOperand in3, CPOperand out, MetaData meta, FileFormatProperties fprops, String schema, UpdateType utype, String sopcode, String istr, int k) { super(CPType.Variable, sopcode, istr); opcode = op; inputs = new ArrayList<>(); @@ -156,76 +132,57 @@ private VariableCPInstruction(VariableOperationCode op, CPOperand in1, CPOperand _formatProperties = fprops; _schema = schema; _updateType = utype; - _containsPreadPrefix = in1 != null && in1.getName() - .contains(org.apache.sysds.lops.Data.PREAD_PREFIX); + _containsPreadPrefix = in1 != null && in1.getName().contains(org.apache.sysds.lops.Data.PREAD_PREFIX); this.k = k; } - private VariableCPInstruction(VariableOperationCode op, CPOperand in1, CPOperand in2, CPOperand in3, CPOperand out, - MetaData meta, FileFormatProperties fprops, String schema, UpdateType utype, String sopcode, String istr) { - this(op ,in1,in2,in3,out,meta, fprops, schema, utype, sopcode, istr, 1); + private VariableCPInstruction(VariableOperationCode op, CPOperand in1, CPOperand in2, CPOperand in3, CPOperand out, MetaData meta, FileFormatProperties fprops, String schema, UpdateType utype, String sopcode, String istr) { + this(op, in1, in2, in3, out, meta, fprops, schema, utype, sopcode, istr, 1); } - private VariableCPInstruction(VariableOperationCode op, CPOperand in1, CPOperand in2, CPOperand in3, CPOperand out, - String sopcode, String istr) { + private VariableCPInstruction(VariableOperationCode op, CPOperand in1, CPOperand in2, CPOperand in3, CPOperand out, String sopcode, String istr) { this(op, in1, in2, in3, out, null, null, null, null, sopcode, istr, 1); } - private VariableCPInstruction(VariableOperationCode op, CPOperand in1, CPOperand in2, CPOperand in3, CPOperand out, - String sopcode, String istr, int k) { + private VariableCPInstruction(VariableOperationCode op, CPOperand in1, CPOperand in2, CPOperand in3, CPOperand out, String sopcode, String istr, int k) { this(op, in1, in2, in3, out, null, null, null, null, sopcode, istr, k); } // This version of the constructor is used only in case of CreateVariable - private VariableCPInstruction(VariableOperationCode op, CPOperand in1, CPOperand in2, CPOperand in3, MetaData md, - UpdateType updateType, String schema, String sopcode, String istr) { + private VariableCPInstruction(VariableOperationCode op, CPOperand in1, CPOperand in2, CPOperand in3, MetaData md, UpdateType updateType, String schema, String sopcode, String istr) { this(op, in1, in2, in3, null, md, null, schema, updateType, sopcode, istr); } // This version of the constructor is used only in case of CreateVariable - private VariableCPInstruction(VariableOperationCode op, CPOperand in1, CPOperand in2, CPOperand in3, MetaData md, - UpdateType updateType, FileFormatProperties formatProperties, String schema, String sopcode, - String istr) { + private VariableCPInstruction(VariableOperationCode op, CPOperand in1, CPOperand in2, CPOperand in3, MetaData md, UpdateType updateType, FileFormatProperties formatProperties, String schema, String sopcode, String istr) { this(op, in1, in2, in3, null, md, formatProperties, schema, updateType, sopcode, istr); } - private static VariableOperationCode getVariableOperationCode ( String str ) { - if ( str.equalsIgnoreCase(Opcodes.CREATEVAR.toString())) - return VariableOperationCode.CreateVariable; - else if ( str.equalsIgnoreCase(Opcodes.ASSIGNVAR.toString())) - return VariableOperationCode.AssignVariable; - else if ( str.equalsIgnoreCase(Opcodes.CPVAR.toString())) - return VariableOperationCode.CopyVariable; - else if ( str.equalsIgnoreCase(Opcodes.MVVAR.toString())) - return VariableOperationCode.MoveVariable; - else if ( str.equalsIgnoreCase(Opcodes.RMVAR.toString()) ) - return VariableOperationCode.RemoveVariable; - else if ( str.equalsIgnoreCase(Opcodes.RMFILEVAR.toString()) ) - return VariableOperationCode.RemoveVariableAndFile; - else if ( str.equalsIgnoreCase(Opcodes.CAST_AS_SCALAR.toString()) ) + private static VariableOperationCode getVariableOperationCode(String str) { + if (str.equalsIgnoreCase(Opcodes.CREATEVAR.toString())) return VariableOperationCode.CreateVariable; + else if (str.equalsIgnoreCase(Opcodes.ASSIGNVAR.toString())) return VariableOperationCode.AssignVariable; + else if (str.equalsIgnoreCase(Opcodes.CPVAR.toString())) return VariableOperationCode.CopyVariable; + else if (str.equalsIgnoreCase(Opcodes.MVVAR.toString())) return VariableOperationCode.MoveVariable; + else if (str.equalsIgnoreCase(Opcodes.RMVAR.toString())) return VariableOperationCode.RemoveVariable; + else if (str.equalsIgnoreCase(Opcodes.RMFILEVAR.toString())) return VariableOperationCode.RemoveVariableAndFile; + else if (str.equalsIgnoreCase(Opcodes.CAST_AS_SCALAR.toString())) return VariableOperationCode.CastAsScalarVariable; - else if ( str.equalsIgnoreCase(Opcodes.CAST_AS_MATRIX.toString()) ) + else if (str.equalsIgnoreCase(Opcodes.CAST_AS_MATRIX.toString())) return VariableOperationCode.CastAsMatrixVariable; - else if ( str.equalsIgnoreCase(Opcodes.CAST_AS_FRAME.toString()) - || str.equalsIgnoreCase(Opcodes.CAST_AS_FRAME_VAR.toString())) + else if (str.equalsIgnoreCase(Opcodes.CAST_AS_FRAME.toString()) || str.equalsIgnoreCase(Opcodes.CAST_AS_FRAME_VAR.toString())) return VariableOperationCode.CastAsFrameVariable; - else if ( str.equalsIgnoreCase(Opcodes.CAST_AS_LIST.toString()) ) - return VariableOperationCode.CastAsListVariable; - else if ( str.equalsIgnoreCase(Opcodes.CAST_AS_DOUBLE.toString()) ) + else if (str.equalsIgnoreCase(Opcodes.CAST_AS_LIST.toString())) return VariableOperationCode.CastAsListVariable; + else if (str.equalsIgnoreCase(Opcodes.CAST_AS_DOUBLE.toString())) return VariableOperationCode.CastAsDoubleVariable; - else if ( str.equalsIgnoreCase(Opcodes.CAST_AS_INT.toString()) ) + else if (str.equalsIgnoreCase(Opcodes.CAST_AS_INT.toString())) return VariableOperationCode.CastAsIntegerVariable; - else if ( str.equalsIgnoreCase(Opcodes.CAST_AS_BOOLEAN.toString()) ) + else if (str.equalsIgnoreCase(Opcodes.CAST_AS_BOOLEAN.toString())) return VariableOperationCode.CastAsBooleanVariable; - else if ( str.equalsIgnoreCase(Opcodes.WRITE.toString()) ) - return VariableOperationCode.Write; - else if ( str.equalsIgnoreCase(Opcodes.READ.toString()) ) - return VariableOperationCode.Read; - else if ( str.equalsIgnoreCase("setfilename") ) - return VariableOperationCode.SetFileName; - else - throw new DMLRuntimeException("Invalid function: " + str); + else if (str.equalsIgnoreCase(Opcodes.WRITE.toString())) return VariableOperationCode.Write; + else if (str.equalsIgnoreCase(Opcodes.READ.toString())) return VariableOperationCode.Read; + else if (str.equalsIgnoreCase("setfilename")) return VariableOperationCode.SetFileName; + else throw new DMLRuntimeException("Invalid function: " + str); } /** @@ -235,10 +192,9 @@ else if ( str.equalsIgnoreCase("setfilename") ) * @return true if rmvar instruction including varName */ public boolean isRemoveVariable(String varName) { - if( isRemoveVariable() ) { - for( CPOperand input : inputs ) - if(input.getName().equalsIgnoreCase(varName)) - return true; + if (isRemoveVariable()) { + for (CPOperand input : inputs) + if (input.getName().equalsIgnoreCase(varName)) return true; } return false; } @@ -248,10 +204,9 @@ public boolean isRemoveVariableNoFile() { } public boolean isRemoveVariable() { - return opcode == VariableOperationCode.RemoveVariable - || opcode == VariableOperationCode.RemoveVariableAndFile; + return opcode == VariableOperationCode.RemoveVariable || opcode == VariableOperationCode.RemoveVariableAndFile; } - + public boolean isMoveVariable() { return opcode == VariableOperationCode.MoveVariable; } @@ -261,8 +216,7 @@ public boolean isAssignVariable() { } public boolean isAssignOrCopyVariable() { - return opcode == VariableOperationCode.AssignVariable - || opcode == VariableOperationCode.CopyVariable; + return opcode == VariableOperationCode.AssignVariable || opcode == VariableOperationCode.CopyVariable; } public boolean isCreateVariable() { @@ -298,31 +252,27 @@ public CPOperand getInput4() { } public CPOperand getInput(int index) { - if( inputs.size() <= index ) - return null; + if (inputs.size() <= index) return null; return inputs.get(index); } public void addInput(CPOperand input) { - if( input != null ) - inputs.add(input); + if (input != null) inputs.add(input); } - public String getOutputVariableName(){ + public String getOutputVariableName() { String ret = null; - if( output != null ) - ret = output.getName(); + if (output != null) ret = output.getName(); return ret; } - public CPOperand getOutput(){ + public CPOperand getOutput() { return output; } private static int getArity(VariableOperationCode op) { - if(op.isCast()) - return 3; - switch(op) { + if (op.isCast()) return 3; + switch (op) { case Write: case SetFileName: return 3; @@ -331,280 +281,251 @@ private static int getArity(VariableOperationCode op) { } } - public static VariableCPInstruction parseInstruction ( String str ) { - String[] parts = InstructionUtils.getInstructionPartsWithValueType ( str ); + public static VariableCPInstruction parseInstruction(String str) { + String[] parts = InstructionUtils.getInstructionPartsWithValueType(str); String opcode = parts[0]; VariableOperationCode voc = getVariableOperationCode(opcode); - - if ( voc == VariableOperationCode.CreateVariable ){ - if ( parts.length < 5 ) //&& parts.length != 10 ) + + if (voc == VariableOperationCode.CreateVariable) { + if (parts.length < 5) //&& parts.length != 10 ) throw new DMLRuntimeException("Invalid number of operands in createvar instruction: " + str); - } - else if ( voc == VariableOperationCode.MoveVariable) { + } else if (voc == VariableOperationCode.MoveVariable) { // mvvar tempA A; or mvvar mvar5 "data/out.mtx" "binary" - if ( parts.length !=3 && parts.length != 4) + if (parts.length != 3 && parts.length != 4) throw new DMLRuntimeException("Invalid number of operands in mvvar instruction: " + str); - } - else if ( voc == VariableOperationCode.Write ) { + } else if (voc == VariableOperationCode.Write) { // All write instructions have 3 parameters, except in case of delimited/csv/libsvm file. // Write instructions for csv files also include three additional parameters (hasHeader, delimiter, sparse) // Write instructions for libsvm files also include one additional parameters (sparse) // TODO - replace hardcoded numbers with more sophisticated code - if ( parts.length != 6 && parts.length != 7 && parts.length != 9 ) + if (parts.length != 6 && parts.length != 7 && parts.length != 9) throw new DMLRuntimeException("Invalid number of operands in write instruction: " + str); - } - else if(voc == VariableOperationCode.CastAsFrameVariable){ + } else if (voc == VariableOperationCode.CastAsFrameVariable) { InstructionUtils.checkNumFields(parts, 3, 4, 5); - } - else { - try{ - if( voc != VariableOperationCode.RemoveVariable ) - InstructionUtils.checkNumFields ( parts, getArity(voc) ); // no output - } - catch(Exception e){ + } else { + try { + if (voc != VariableOperationCode.RemoveVariable) + InstructionUtils.checkNumFields(parts, getArity(voc)); // no output + } catch (Exception e) { throw new DMLRuntimeException("Invalid number of fields with operation code: " + voc, e); } } - CPOperand in1=null, in2=null, in3=null, in4=null, out=null; + CPOperand in1 = null, in2 = null, in3 = null, in4 = null, out = null; int k = 1; switch (voc) { - case CreateVariable: - // variable name - DataType dt = DataType.valueOf(parts[4]); - //TODO choose correct value type for tensor - ValueType vt = dt==DataType.MATRIX ? ValueType.FP64 : ValueType.STRING; - int extSchema = (dt==DataType.FRAME && parts.length>=12) ? 1 : 0; - in1 = new CPOperand(parts[1], vt, dt); - // file name - in2 = new CPOperand(parts[2], ValueType.STRING, DataType.SCALAR); - // file name override flag (always literal) - in3 = new CPOperand(parts[3], ValueType.BOOLEAN, DataType.SCALAR); - - // format - String fmt = parts[5]; - if ( fmt.equalsIgnoreCase("csv") ) { - // Cretevar instructions for CSV format either has 13 or 14 inputs. - // 13 inputs: createvar corresponding to WRITE -- includes properties hasHeader, delim, and sparse - // 14 inputs: createvar corresponding to READ -- includes properties hasHeader, delim, fill, and fillValue - if ( parts.length < 14+extSchema || parts.length > 16+extSchema ) - throw new DMLRuntimeException("Invalid number of operands in createvar instruction: " + str); - } - else if(fmt.equalsIgnoreCase("libsvm")) { - // 13 inputs: createvar corresponding to WRITE -- includes properties delim, index delim, and sparse - // 12 inputs: createvar corresponding to READ -- includes properties delim, index delim, and sparse - - if(parts.length < 12 + extSchema) - throw new DMLRuntimeException("Invalid number of operands in createvar instruction: " + str); - } - else if(fmt.equalsIgnoreCase("hdf5")) { - // 11 inputs: createvar corresponding to WRITE/READ -- includes properties dataset name - if(parts.length < 11 + extSchema) - throw new DMLRuntimeException("Invalid number of operands in createvar instruction: " + str); - } - else { - if ( parts.length != 6 && parts.length != 11+extSchema ) - throw new DMLRuntimeException("Invalid number of operands in createvar instruction: " + str); - } - - MetaDataFormat iimd = null; - if (dt == DataType.MATRIX || dt == DataType.FRAME || dt == DataType.LIST) { - DataCharacteristics mc = new MatrixCharacteristics(); - if (parts.length == 6) { - // do nothing - } - else if (parts.length >= 10) { - // matrix characteristics - mc.setDimension(Long.parseLong(parts[6]), Long.parseLong(parts[7])); - mc.setBlocksize(Integer.parseInt(parts[8])); - mc.setNonZeros(Long.parseLong(parts[9])); - } - else { - throw new DMLRuntimeException("Invalid number of operands in createvar instruction: " + str); - } - iimd = new MetaDataFormat(mc, FileFormat.safeValueOf(fmt)); - } - else if (dt == DataType.TENSOR) { - TensorCharacteristics tc = new TensorCharacteristics(new long[]{1, 1}, 0); - if (parts.length == 6) { - // do nothing - } - else if (parts.length >= 10) { - // TODO correct sizes - tc.setDim(0, Long.parseLong(parts[6])); - tc.setDim(1, Long.parseLong(parts[7])); - tc.setBlocksize(Integer.parseInt(parts[8])); - } - else { - throw new DMLRuntimeException("Invalid number of operands in createvar instruction: " + str); - } - iimd = new MetaDataFormat(tc, FileFormat.safeValueOf(fmt)); - } - UpdateType updateType = UpdateType.COPY; - if ( parts.length >= 11 ) - updateType = UpdateType.valueOf(parts[10].toUpperCase()); - - //handle frame schema - String schema = (dt==DataType.FRAME && parts.length>=12) ? parts[parts.length-1] : null; - - if ( fmt.equalsIgnoreCase("csv") ) { - // Cretevar instructions for CSV format either has 13 or 14 inputs. - // 13 inputs: createvar corresponding to WRITE -- includes properties hasHeader, delim, and sparse - // 14 inputs: createvar corresponding to READ -- includes properties hasHeader, delim, fill, and fillValue - FileFormatProperties fmtProperties = null; - int curPos = 11; - if ( parts.length == 14+extSchema ) { - boolean hasHeader = Boolean.parseBoolean(parts[curPos]); - String delim = parts[curPos+1]; - boolean sparse = Boolean.parseBoolean(parts[curPos+2]); - fmtProperties = new FileFormatPropertiesCSV(hasHeader, delim, sparse) ; + case CreateVariable: + // variable name + DataType dt = DataType.valueOf(parts[4]); + //TODO choose correct value type for tensor + ValueType vt = dt == DataType.MATRIX ? ValueType.FP64 : ValueType.STRING; + int extSchema = (dt == DataType.FRAME && parts.length >= 12) ? 1 : 0; + in1 = new CPOperand(parts[1], vt, dt); + // file name + in2 = new CPOperand(parts[2], ValueType.STRING, DataType.SCALAR); + // file name override flag (always literal) + in3 = new CPOperand(parts[3], ValueType.BOOLEAN, DataType.SCALAR); + + // format + String fmt = parts[5]; + if (fmt.equalsIgnoreCase("csv")) { + // Cretevar instructions for CSV format either has 13 or 14 inputs. + // 13 inputs: createvar corresponding to WRITE -- includes properties hasHeader, delim, and sparse + // 14 inputs: createvar corresponding to READ -- includes properties hasHeader, delim, fill, and fillValue + if (parts.length < 14 + extSchema || parts.length > 16 + extSchema) + throw new DMLRuntimeException("Invalid number of operands in createvar instruction: " + str); + } else if (fmt.equalsIgnoreCase("libsvm")) { + // 13 inputs: createvar corresponding to WRITE -- includes properties delim, index delim, and sparse + // 12 inputs: createvar corresponding to READ -- includes properties delim, index delim, and sparse + + if (parts.length < 12 + extSchema) + throw new DMLRuntimeException("Invalid number of operands in createvar instruction: " + str); + } else if (fmt.equalsIgnoreCase("hdf5")) { + // 11 inputs: createvar corresponding to WRITE/READ -- includes properties dataset name + if (parts.length < 11 + extSchema) + throw new DMLRuntimeException("Invalid number of operands in createvar instruction: " + str); + } else { + if (parts.length != 6 && parts.length != 11 + extSchema) + throw new DMLRuntimeException("Invalid number of operands in createvar instruction: " + str); } - else { - boolean hasHeader = Boolean.parseBoolean(parts[curPos]); - String delim = parts[curPos+1]; - boolean fill = Boolean.parseBoolean(parts[curPos+2]); - double fillValue = Double.parseDouble(parts[curPos+3]); - String naStrings = null; - if ( parts.length == 16+extSchema ) - naStrings = parts[curPos+4]; - fmtProperties = new FileFormatPropertiesCSV(hasHeader, delim, fill, fillValue, naStrings) ; + + MetaDataFormat iimd = null; + if (dt == DataType.MATRIX || dt == DataType.FRAME || dt == DataType.LIST) { + DataCharacteristics mc = new MatrixCharacteristics(); + if (parts.length == 6) { + // do nothing + } else if (parts.length >= 10) { + // matrix characteristics + mc.setDimension(Long.parseLong(parts[6]), Long.parseLong(parts[7])); + mc.setBlocksize(Integer.parseInt(parts[8])); + mc.setNonZeros(Long.parseLong(parts[9])); + } else { + throw new DMLRuntimeException("Invalid number of operands in createvar instruction: " + str); + } + iimd = new MetaDataFormat(mc, FileFormat.safeValueOf(fmt)); + } else if (dt == DataType.TENSOR) { + TensorCharacteristics tc = new TensorCharacteristics(new long[]{1, 1}, 0); + if (parts.length == 6) { + // do nothing + } else if (parts.length >= 10) { + // TODO correct sizes + tc.setDim(0, Long.parseLong(parts[6])); + tc.setDim(1, Long.parseLong(parts[7])); + tc.setBlocksize(Integer.parseInt(parts[8])); + } else { + throw new DMLRuntimeException("Invalid number of operands in createvar instruction: " + str); + } + iimd = new MetaDataFormat(tc, FileFormat.safeValueOf(fmt)); } - return new VariableCPInstruction(VariableOperationCode.CreateVariable, - in1, in2, in3, iimd, updateType, fmtProperties, schema, opcode, str); - } - else if(fmt.equalsIgnoreCase("libsvm")) { - // Cretevar instructions for LIBSVM format has 13. - // 13 inputs: createvar corresponding to WRITE -- includes properties delim, index delim and sparse - // 12 inputs: createvar corresponding to READ -- includes properties delim, index delim, and sparse - FileFormatProperties fmtProperties = null; - int curPos = 11; - if(parts.length == 12 + extSchema) { - String delim = parts[curPos]; - String indexDelim = parts[curPos + 1]; - fmtProperties = new FileFormatPropertiesLIBSVM(delim, indexDelim); + UpdateType updateType = UpdateType.COPY; + if (parts.length >= 11) updateType = UpdateType.valueOf(parts[10].toUpperCase()); + + //handle frame schema + String schema = (dt == DataType.FRAME && parts.length >= 12) ? parts[parts.length - 1] : null; + + if (fmt.equalsIgnoreCase("csv")) { + // Cretevar instructions for CSV format either has 13 or 14 inputs. + // 13 inputs: createvar corresponding to WRITE -- includes properties hasHeader, delim, and sparse + // 14 inputs: createvar corresponding to READ -- includes properties hasHeader, delim, fill, and fillValue + FileFormatProperties fmtProperties = null; + int curPos = 11; + if (parts.length == 14 + extSchema) { + boolean hasHeader = Boolean.parseBoolean(parts[curPos]); + String delim = parts[curPos + 1]; + boolean sparse = Boolean.parseBoolean(parts[curPos + 2]); + fmtProperties = new FileFormatPropertiesCSV(hasHeader, delim, sparse); + } else { + boolean hasHeader = Boolean.parseBoolean(parts[curPos]); + String delim = parts[curPos + 1]; + boolean fill = Boolean.parseBoolean(parts[curPos + 2]); + double fillValue = Double.parseDouble(parts[curPos + 3]); + String naStrings = null; + if (parts.length == 16 + extSchema) naStrings = parts[curPos + 4]; + fmtProperties = new FileFormatPropertiesCSV(hasHeader, delim, fill, fillValue, naStrings); + } + return new VariableCPInstruction(VariableOperationCode.CreateVariable, in1, in2, in3, iimd, updateType, fmtProperties, schema, opcode, str); + } else if (fmt.equalsIgnoreCase("libsvm")) { + // Cretevar instructions for LIBSVM format has 13. + // 13 inputs: createvar corresponding to WRITE -- includes properties delim, index delim and sparse + // 12 inputs: createvar corresponding to READ -- includes properties delim, index delim, and sparse + FileFormatProperties fmtProperties = null; + int curPos = 11; + if (parts.length == 12 + extSchema) { + String delim = parts[curPos]; + String indexDelim = parts[curPos + 1]; + fmtProperties = new FileFormatPropertiesLIBSVM(delim, indexDelim); + } else { + String delim = parts[curPos]; + String indexDelim = parts[curPos + 1]; + boolean sparse = Boolean.parseBoolean(parts[curPos + 2]); + fmtProperties = new FileFormatPropertiesLIBSVM(delim, indexDelim, sparse); + } + + return new VariableCPInstruction(VariableOperationCode.CreateVariable, in1, in2, in3, iimd, updateType, fmtProperties, schema, opcode, str); + } else if (fmt.equalsIgnoreCase("hdf5")) { + // Cretevar instructions for HDF5 format has 13. + // 11 inputs: createvar corresponding to WRITE/READ -- includes properties dataset name + int curPos = 11; + String datasetName = parts[curPos]; + FileFormatProperties fmtProperties = new FileFormatPropertiesHDF5(datasetName); + + return new VariableCPInstruction(VariableOperationCode.CreateVariable, in1, in2, in3, iimd, updateType, fmtProperties, schema, opcode, str); + } else { + return new VariableCPInstruction(VariableOperationCode.CreateVariable, in1, in2, in3, iimd, updateType, schema, opcode, str); } - else { - String delim = parts[curPos]; - String indexDelim = parts[curPos + 1]; - boolean sparse = Boolean.parseBoolean(parts[curPos + 2]); - fmtProperties = new FileFormatPropertiesLIBSVM(delim, indexDelim, sparse); + + case AssignVariable: + in1 = new CPOperand(parts[1]); + in2 = new CPOperand(parts[2]); + break; + + case CopyVariable: + // Value types are not given here + boolean withTypes = parts[1].split(VALUETYPE_PREFIX).length > 2 && parts[2].split(VALUETYPE_PREFIX).length > 2; + in1 = withTypes ? new CPOperand(parts[1]) : new CPOperand(parts[1], ValueType.UNKNOWN, DataType.UNKNOWN); + in2 = withTypes ? new CPOperand(parts[2]) : new CPOperand(parts[2], ValueType.UNKNOWN, DataType.UNKNOWN); + break; + + case MoveVariable: + in1 = new CPOperand(parts[1], ValueType.UNKNOWN, DataType.UNKNOWN); + in2 = new CPOperand(parts[2], ValueType.UNKNOWN, DataType.UNKNOWN); + if (parts.length > 3) in3 = new CPOperand(parts[3], ValueType.UNKNOWN, DataType.UNKNOWN); + break; + + case RemoveVariable: + VariableCPInstruction rminst = new VariableCPInstruction(getVariableOperationCode(opcode), null, null, null, out, opcode, str); + for (int i = 1; i < parts.length; i++) + rminst.addInput(new CPOperand(parts[i], ValueType.UNKNOWN, DataType.SCALAR)); + return rminst; + + case RemoveVariableAndFile: + in1 = new CPOperand(parts[1]); + in2 = new CPOperand(parts[2]); + // second argument must be a boolean + if (in2.getValueType() != ValueType.BOOLEAN) + throw new DMLRuntimeException("Unexpected value type for second argument in: " + str); + break; + + case CastAsFrameVariable: + if (parts.length == 5) { + in1 = new CPOperand(parts[1]); // input to cast + in2 = new CPOperand(parts[2]); // list of column names + out = new CPOperand(parts[3]); // output + k = Integer.parseInt(parts[4]); + break; } - - return new VariableCPInstruction(VariableOperationCode.CreateVariable, - in1, in2, in3, iimd, updateType, fmtProperties, schema, opcode, str); - } - else if(fmt.equalsIgnoreCase("hdf5")) { - // Cretevar instructions for HDF5 format has 13. - // 11 inputs: createvar corresponding to WRITE/READ -- includes properties dataset name - int curPos = 11; - String datasetName = parts[curPos]; - FileFormatProperties fmtProperties = new FileFormatPropertiesHDF5(datasetName); - - return new VariableCPInstruction(VariableOperationCode.CreateVariable, - in1, in2, in3, iimd, updateType, fmtProperties, schema, opcode, str); - } - else { - return new VariableCPInstruction(VariableOperationCode.CreateVariable, in1, in2, in3, iimd, updateType, schema, opcode, str); - } - - case AssignVariable: - in1 = new CPOperand(parts[1]); - in2 = new CPOperand(parts[2]); - break; - - case CopyVariable: - // Value types are not given here - boolean withTypes = parts[1].split(VALUETYPE_PREFIX).length > 2 && parts[2].split(VALUETYPE_PREFIX).length > 2; - in1 = withTypes ? new CPOperand(parts[1]) : new CPOperand(parts[1], ValueType.UNKNOWN, DataType.UNKNOWN); - in2 = withTypes ? new CPOperand(parts[2]) : new CPOperand(parts[2], ValueType.UNKNOWN, DataType.UNKNOWN); - break; - - case MoveVariable: - in1 = new CPOperand(parts[1], ValueType.UNKNOWN, DataType.UNKNOWN); - in2 = new CPOperand(parts[2], ValueType.UNKNOWN, DataType.UNKNOWN); - if(parts.length > 3) - in3 = new CPOperand(parts[3], ValueType.UNKNOWN, DataType.UNKNOWN); - break; - - case RemoveVariable: - VariableCPInstruction rminst = new VariableCPInstruction( - getVariableOperationCode(opcode), null, null, null, out, opcode, str); - for( int i=1; i string value type + out = new CPOperand(parts[2]); // output variable name + k = Integer.parseInt(parts[3]); // thread count break; - } - case CastAsScalarVariable: - case CastAsMatrixVariable: - case CastAsListVariable: - case CastAsDoubleVariable: - case CastAsIntegerVariable: - case CastAsBooleanVariable: - in1 = new CPOperand(parts[1]); // first operand is a variable name => string value type - out = new CPOperand(parts[2]); // output variable name - k = Integer.parseInt(parts[3]); // thread count - break; - - case Write: - in1 = new CPOperand(parts[1]); - in2 = new CPOperand(parts[2]); - in3 = new CPOperand(parts[3]); - - FileFormatProperties fprops = null; - if ( in3.getName().equalsIgnoreCase("csv") ) { - boolean hasHeader = Boolean.parseBoolean(parts[4]); - String delim = parts[5]; - boolean sparse = Boolean.parseBoolean(parts[6]); - fprops = new FileFormatPropertiesCSV(hasHeader, delim, sparse); - in4 = new CPOperand(parts[7]); // description - } - else if ( in3.getName().equalsIgnoreCase("libsvm") ) { - String delim = parts[4]; - String indexDelim = parts[5]; - boolean sparse = Boolean.parseBoolean(parts[6]); - fprops = new FileFormatPropertiesLIBSVM(delim, indexDelim, sparse); - } - else if(in3.getName().equalsIgnoreCase("hdf5") ){ - String datasetName = parts[4]; - fprops = new FileFormatPropertiesHDF5(datasetName); - } - else { - fprops = new FileFormatProperties(); - in4 = new CPOperand(parts[5]); // blocksize in empty description - } - VariableCPInstruction inst = new VariableCPInstruction( - getVariableOperationCode(opcode), in1, in2, in3, out, null, fprops, null, null, opcode, str); - inst.addInput(in4); - return inst; + case Write: + in1 = new CPOperand(parts[1]); + in2 = new CPOperand(parts[2]); + in3 = new CPOperand(parts[3]); + + FileFormatProperties fprops = null; + if (in3.getName().equalsIgnoreCase("csv")) { + boolean hasHeader = Boolean.parseBoolean(parts[4]); + String delim = parts[5]; + boolean sparse = Boolean.parseBoolean(parts[6]); + fprops = new FileFormatPropertiesCSV(hasHeader, delim, sparse); + in4 = new CPOperand(parts[7]); // description + } else if (in3.getName().equalsIgnoreCase("libsvm")) { + String delim = parts[4]; + String indexDelim = parts[5]; + boolean sparse = Boolean.parseBoolean(parts[6]); + fprops = new FileFormatPropertiesLIBSVM(delim, indexDelim, sparse); + } else if (in3.getName().equalsIgnoreCase("hdf5")) { + String datasetName = parts[4]; + fprops = new FileFormatPropertiesHDF5(datasetName); + } else { + fprops = new FileFormatProperties(); + in4 = new CPOperand(parts[5]); // blocksize in empty description + } + VariableCPInstruction inst = new VariableCPInstruction(getVariableOperationCode(opcode), in1, in2, in3, out, null, fprops, null, null, opcode, str); + inst.addInput(in4); - case Read: - in1 = new CPOperand(parts[1]); - in2 = new CPOperand(parts[2]); - break; + return inst; - case SetFileName: - in1 = new CPOperand(parts[1]); // variable name - in2 = new CPOperand(parts[2], ValueType.UNKNOWN, DataType.UNKNOWN); // file name - in3 = new CPOperand(parts[3], ValueType.UNKNOWN, DataType.UNKNOWN); // option: remote or local - break; + case Read: + in1 = new CPOperand(parts[1]); + in2 = new CPOperand(parts[2]); + break; + + case SetFileName: + in1 = new CPOperand(parts[1]); // variable name + in2 = new CPOperand(parts[2], ValueType.UNKNOWN, DataType.UNKNOWN); // file name + in3 = new CPOperand(parts[3], ValueType.UNKNOWN, DataType.UNKNOWN); // option: remote or local + break; } return new VariableCPInstruction(getVariableOperationCode(opcode), in1, in2, in3, out, opcode, str, k); @@ -612,84 +533,83 @@ else if(in3.getName().equalsIgnoreCase("hdf5") ){ @Override public void processInstruction(ExecutionContext ec) { - switch ( opcode ) - { - case CreateVariable: - processCreateVariableInstruction(ec); - break; - - case AssignVariable: - // assign value of variable to the other - ec.setScalarOutput(getInput2().getName(), ec.getScalarInput(getInput1())); - break; - - case CopyVariable: - processCopyInstruction(ec); - break; - - case MoveVariable: - processMoveInstruction(ec); - break; - - case RemoveVariable: - for( CPOperand input : inputs ) - processRmvarInstruction(ec, input.getName()); - break; - - case RemoveVariableAndFile: - processRemoveVariableAndFileInstruction(ec); - break; - - case CastAsScalarVariable: //castAsScalarVariable - processCastAsScalarVariableInstruction(ec); - break; - - case CastAsMatrixVariable: - processCastAsMatrixVariableInstruction(ec); - break; - - case CastAsFrameVariable: - processCastAsFrameVariableInstruction(ec); - break; - - case CastAsListVariable: - ListObject lobj = ec.getListObject(getInput1()); - if( lobj.getLength() != 1 || !(lobj.getData(0) instanceof ListObject) ) - ec.setVariable(output.getName(), lobj); + switch (opcode) { + case CreateVariable: + processCreateVariableInstruction(ec); + break; + + case AssignVariable: + // assign value of variable to the other + ec.setScalarOutput(getInput2().getName(), ec.getScalarInput(getInput1())); + break; + + case CopyVariable: + processCopyInstruction(ec); + break; + + case MoveVariable: + processMoveInstruction(ec); + break; + + case RemoveVariable: + for (CPOperand input : inputs) + processRmvarInstruction(ec, input.getName()); + break; + + case RemoveVariableAndFile: + processRemoveVariableAndFileInstruction(ec); + break; + + case CastAsScalarVariable: //castAsScalarVariable + processCastAsScalarVariableInstruction(ec); + break; + + case CastAsMatrixVariable: + processCastAsMatrixVariableInstruction(ec); + break; + + case CastAsFrameVariable: + processCastAsFrameVariableInstruction(ec); + break; + + case CastAsListVariable: + ListObject lobj = ec.getListObject(getInput1()); + if (lobj.getLength() != 1 || !(lobj.getData(0) instanceof ListObject)) + ec.setVariable(output.getName(), lobj); // throw new RuntimeException("as.list() expects a list input with one nested list: " // + "length(list)="+lobj.getLength()+", dt(list[0])="+lobj.getData(0).getDataType() ); - else ec.setVariable(output.getName(), lobj.getData(0)); - break; - - case CastAsDoubleVariable: - ScalarObject scalarDoubleInput = ec.getScalarInput(getInput1()); - ec.setScalarOutput(output.getName(), ScalarObjectFactory.castToDouble(scalarDoubleInput)); - break; - - case CastAsIntegerVariable: - ScalarObject scalarLongInput = ec.getScalarInput(getInput1()); - ec.setScalarOutput(output.getName(), ScalarObjectFactory.castToLong(scalarLongInput)); - break; - - case CastAsBooleanVariable: - ScalarObject scalarBooleanInput = ec.getScalarInput(getInput1()); - ec.setScalarOutput(output.getName(), new BooleanObject(scalarBooleanInput.getBooleanValue())); - break; - - case Read: - processReadInstruction(ec); - break; - - case Write: - processWriteInstruction(ec); - break; - - case SetFileName: - processSetFileNameInstruction(ec); - break; - - default: - throw new DMLRuntimeException("Unknown opcode: " + opcode ); + else ec.setVariable(output.getName(), lobj.getData(0)); + break; + + case CastAsDoubleVariable: + ScalarObject scalarDoubleInput = ec.getScalarInput(getInput1()); + ec.setScalarOutput(output.getName(), ScalarObjectFactory.castToDouble(scalarDoubleInput)); + break; + + case CastAsIntegerVariable: + ScalarObject scalarLongInput = ec.getScalarInput(getInput1()); + ec.setScalarOutput(output.getName(), ScalarObjectFactory.castToLong(scalarLongInput)); + break; + + case CastAsBooleanVariable: + ScalarObject scalarBooleanInput = ec.getScalarInput(getInput1()); + ec.setScalarOutput(output.getName(), new BooleanObject(scalarBooleanInput.getBooleanValue())); + break; + + case Read: + processReadInstruction(ec); + break; + + case Write: + processWriteInstruction(ec); + break; + + case SetFileName: + processSetFileNameInstruction(ec); + break; + + default: + throw new DMLRuntimeException("Unknown opcode: " + opcode); } } @@ -698,13 +618,12 @@ public void processInstruction(ExecutionContext ec) { * * @param ec execution context of the instruction */ - private void processCreateVariableInstruction(ExecutionContext ec){ + private void processCreateVariableInstruction(ExecutionContext ec) { //PRE: for robustness we cleanup existing variables, because a setVariable //would cause a buffer pool memory leak as these objects would never be removed - if(ec.containsVariable(getInput1())) - processRmvarInstruction(ec, getInput1().getName()); + if (ec.containsVariable(getInput1())) processRmvarInstruction(ec, getInput1().getName()); - switch(getInput1().getDataType()) { + switch (getInput1().getDataType()) { case MATRIX: { String fname = createUniqueFilename(); MatrixObject obj = new MatrixObject(getInput1().getValueType(), fname); @@ -712,8 +631,7 @@ private void processCreateVariableInstruction(ExecutionContext ec){ obj.setUpdateType(_updateType); obj.setMarkForLinCache(true); ec.setVariable(getInput1().getName(), obj); - if(DMLScript.STATISTICS && _updateType.isInPlace()) - Statistics.incrementTotalUIPVar(); + if (DMLScript.STATISTICS && _updateType.isInPlace()) Statistics.incrementTotalUIPVar(); break; } case TENSOR: { @@ -726,15 +644,33 @@ private void processCreateVariableInstruction(ExecutionContext ec){ case FRAME: { String fname = createUniqueFilename(); FrameObject fobj = new FrameObject(fname); - setCacheableDataFields(fobj, getInput1().getName()); - if( _schema != null ) - fobj.setSchema(_schema); //after metadata - ec.setVariable(getInput1().getName(), fobj); + + String inputName = getInput1().getName(); + setCacheableDataFields(fobj, inputName); + + if (_schema != null) fobj.setSchema(_schema); + + if (_formatProperties instanceof FileFormatPropertiesCSV) { + FileFormatPropertiesCSV props = (FileFormatPropertiesCSV) _formatProperties; + + if (props.hasHeader()) { + FrameReaderTextCSV reader = new FrameReaderTextCSV(props); + + String[] names = reader.readColumnNamesFromHDFS(fname); + + fobj.setColumnNames(names); + + if (fobj.getColumnNames() == null) { + throw new DMLRuntimeException("Column names were not stored in FrameObject!"); + } + } + } + + ec.setVariable(inputName, fobj); break; } case LIST: { - ListObject lo = ListReader.readListFromHDFS(getInput2().getName(), - ((MetaDataFormat)metadata).getFileFormat().name(), _formatProperties); + ListObject lo = ListReader.readListFromHDFS(getInput2().getName(), ((MetaDataFormat) metadata).getFileFormat().name(), _formatProperties); ec.setVariable(getInput1().getName(), lo); break; } @@ -748,23 +684,22 @@ private void processCreateVariableInstruction(ExecutionContext ec){ } } - private String createUniqueFilename(){ + private String createUniqueFilename() { //create new variable for symbol table and cache //(existing objects gets cleared through rmvar instructions) String fname = getInput2().getName(); // check if unique filename needs to be generated - if( Boolean.parseBoolean(getInput3().getName()) ) { + if (Boolean.parseBoolean(getInput3().getName())) { fname = getUniqueFileName(fname); } return fname; } - private void setCacheableDataFields(CacheableData obj, String varname){ + private void setCacheableDataFields(CacheableData obj, String varname) { //clone metadata because it is updated on copy-on-write, otherwise there //is potential for hidden side effects between variables. - obj.setMetaData((MetaData)metadata.clone()); - obj.enableCleanup(!getInput1().getName() - .startsWith(org.apache.sysds.lops.Data.PREAD_PREFIX)); + obj.setMetaData((MetaData) metadata.clone()); + obj.enableCleanup(!getInput1().getName().startsWith(org.apache.sysds.lops.Data.PREAD_PREFIX)); obj.setFileFormatProperties(_formatProperties); obj.setPersistentRead(varname.startsWith(org.apache.sysds.lops.Data.PREAD_PREFIX)); } @@ -780,53 +715,46 @@ private void setCacheableDataFields(CacheableData obj, String varname){ @SuppressWarnings("rawtypes") private void processMoveInstruction(ExecutionContext ec) { - if ( getInput3() == null ) { + if (getInput3() == null) { // example: mvvar tempA A (note that mvvar does not carry the data types) // get and remove source variable Data srcData = ec.removeVariable(getInput1().getName()); - if ( srcData == null ) { - throw new DMLRuntimeException("Unexpected error: could not find a data object " - + "for variable name: " + getInput1().getName() + ", while processing instruction "); + if (srcData == null) { + throw new DMLRuntimeException("Unexpected error: could not find a data object " + "for variable name: " + getInput1().getName() + ", while processing instruction "); } // remove existing variable bound to target name and // cleanup matrix/frame/list data if necessary - if( srcData.getDataType().isMatrix() || srcData.getDataType().isFrame() ) { + if (srcData.getDataType().isMatrix() || srcData.getDataType().isFrame()) { Data tgtData = ec.removeVariable(getInput2().getName()); if (DMLScript.USE_OOC && tgtData instanceof MatrixObject) TeeOOCInstruction.incrRef(((MatrixObject) tgtData).getStreamable(), -1); - if( tgtData != null && srcData != tgtData ) - ec.cleanupDataObject(tgtData); + if (tgtData != null && srcData != tgtData) ec.cleanupDataObject(tgtData); } // do the actual move ec.setVariable(getInput2().getName(), srcData); - } - else { + } else { // example instruction: mvvar - if ( ec.getVariable(getInput1().getName()) == null ) - throw new DMLRuntimeException("Unexpected error: could not find a data object for variable name:" + getInput1().getName() + ", while processing instruction " +this.toString()); + if (ec.getVariable(getInput1().getName()) == null) + throw new DMLRuntimeException("Unexpected error: could not find a data object for variable name:" + getInput1().getName() + ", while processing instruction " + this.toString()); Data object = ec.getVariable(getInput1().getName()); - if ( getInput3().getName().equalsIgnoreCase("binaryblock") ) { + if (getInput3().getName().equalsIgnoreCase("binaryblock")) { boolean success = false; - success = ((CacheableData)object).moveData(getInput2().getName(), getInput3().getName()); + success = ((CacheableData) object).moveData(getInput2().getName(), getInput3().getName()); if (!success) { throw new DMLRuntimeException("Failed to move var " + getInput1().getName() + " to file " + getInput2().getName() + "."); } - } - else - if(object instanceof MatrixObject) - throw new DMLRuntimeException("Unexpected formats while copying: from matrix blocks [" - + ((MatrixObject)object).getBlocksize() + "] to " + getInput3().getName()); - else if (object instanceof FrameObject) - throw new DMLRuntimeException("Unexpected formats while copying: from fram object [" - + ((FrameObject)object).getNumColumns() + "," + ((FrameObject)object).getNumColumns() + "] to " + getInput3().getName()); + } else if (object instanceof MatrixObject) + throw new DMLRuntimeException("Unexpected formats while copying: from matrix blocks [" + ((MatrixObject) object).getBlocksize() + "] to " + getInput3().getName()); + else if (object instanceof FrameObject) + throw new DMLRuntimeException("Unexpected formats while copying: from fram object [" + ((FrameObject) object).getNumColumns() + "," + ((FrameObject) object).getNumColumns() + "] to " + getInput3().getName()); } } @@ -835,25 +763,23 @@ else if (object instanceof FrameObject) * * @param ec execution context */ - private void processRemoveVariableAndFileInstruction(ExecutionContext ec){ + private void processRemoveVariableAndFileInstruction(ExecutionContext ec) { // Remove the variable from HashMap _variables, and possibly delete the data on disk. - boolean del = ( (BooleanObject) ec.getScalarInput(getInput2().getName(), getInput2().getValueType(), true) ).getBooleanValue(); + boolean del = ((BooleanObject) ec.getScalarInput(getInput2().getName(), getInput2().getValueType(), true)).getBooleanValue(); MatrixObject m = (MatrixObject) ec.removeVariable(getInput1().getName()); - if ( !del ) { + if (!del) { // HDFS file should be retailed after clearData(), // therefore data must be exported if dirty flag is set - if ( m.isDirty() ) - m.exportData(); - } - else { + if (m.isDirty()) m.exportData(); + } else { //throw new DMLRuntimeException("rmfilevar w/ true is not expected! " + instString); //cleanDataOnHDFS(pb, input1.getName()); - cleanDataOnHDFS( m ); + cleanDataOnHDFS(m); } // check if in-memory object can be cleaned up - if ( !ec.getVariables().hasReferences(m) ) { + if (!ec.getVariables().hasReferences(m)) { // no other variable in the symbol table points to the same Data object as that of input1.getName() //remove matrix object from cache @@ -863,16 +789,16 @@ private void processRemoveVariableAndFileInstruction(ExecutionContext ec){ /** * Process CastAsScalarVariable instruction. - * + * * @param ec execution context */ - private void processCastAsScalarVariableInstruction(ExecutionContext ec){ + private void processCastAsScalarVariableInstruction(ExecutionContext ec) { - switch( getInput1().getDataType() ) { + switch (getInput1().getDataType()) { case MATRIX: { MatrixBlock mBlock = ec.getMatrixInput(getInput1().getName()); - if( mBlock.getNumRows()!=1 || mBlock.getNumColumns()!=1 ) - throw new DMLRuntimeException("Dimension mismatch - unable to cast matrix '"+getInput1().getName()+"' of dimension ("+mBlock.getNumRows()+" x "+mBlock.getNumColumns()+") to scalar. "); + if (mBlock.getNumRows() != 1 || mBlock.getNumColumns() != 1) + throw new DMLRuntimeException("Dimension mismatch - unable to cast matrix '" + getInput1().getName() + "' of dimension (" + mBlock.getNumRows() + " x " + mBlock.getNumColumns() + ") to scalar. "); double value = mBlock.get(0, 0); ec.releaseMatrixInput(getInput1().getName()); ec.setScalarOutput(output.getName(), new DoubleObject(value)); @@ -880,12 +806,11 @@ private void processCastAsScalarVariableInstruction(ExecutionContext ec){ } case FRAME: { FrameBlock fBlock = ec.getFrameInput(getInput1().getName()); - if( fBlock.getNumRows()!=1 || fBlock.getNumColumns()!=1 ) - throw new DMLRuntimeException("Dimension mismatch - unable to cast frame '"+getInput1().getName()+"' of dimension ("+fBlock.getNumRows()+" x "+fBlock.getNumColumns()+") to scalar."); - Object value = fBlock.get(0,0); + if (fBlock.getNumRows() != 1 || fBlock.getNumColumns() != 1) + throw new DMLRuntimeException("Dimension mismatch - unable to cast frame '" + getInput1().getName() + "' of dimension (" + fBlock.getNumRows() + " x " + fBlock.getNumColumns() + ") to scalar."); + Object value = fBlock.get(0, 0); ec.releaseFrameInput(getInput1().getName()); - ec.setScalarOutput(output.getName(), - ScalarObjectFactory.createScalarObject(fBlock.getSchema()[0], value)); + ec.setScalarOutput(output.getName(), ScalarObjectFactory.createScalarObject(fBlock.getSchema()[0], value)); break; } case TENSOR: { @@ -893,14 +818,13 @@ private void processCastAsScalarVariableInstruction(ExecutionContext ec){ if (tBlock.getNumDims() != 2 || tBlock.getNumRows() != 1 || tBlock.getNumColumns() != 1) throw new DMLRuntimeException("Dimension mismatch - unable to cast tensor '" + getInput1().getName() + "' to scalar."); ValueType vt = !tBlock.isBasic() ? tBlock.getSchema()[0] : tBlock.getValueType(); - ec.setScalarOutput(output.getName(), ScalarObjectFactory - .createScalarObject(vt, tBlock.get(new int[] {0, 0}))); + ec.setScalarOutput(output.getName(), ScalarObjectFactory.createScalarObject(vt, tBlock.get(new int[]{0, 0}))); ec.releaseTensorInput(getInput1().getName()); break; } case LIST: { //TODO handling of cleanup status, potentially new object - ListObject list = (ListObject)ec.getVariable(getInput1().getName()); + ListObject list = (ListObject) ec.getVariable(getInput1().getName()); ec.setVariable(output.getName(), list.slice(0)); break; } @@ -910,8 +834,7 @@ private void processCastAsScalarVariableInstruction(ExecutionContext ec){ break; } default: - throw new DMLRuntimeException("Unsupported data type " - + "in as.scalar(): "+getInput1().getDataType().name()); + throw new DMLRuntimeException("Unsupported data type " + "in as.scalar(): " + getInput1().getDataType().name()); } } @@ -921,7 +844,7 @@ private void processCastAsScalarVariableInstruction(ExecutionContext ec){ * @param ec execution context */ private void processCastAsMatrixVariableInstruction(ExecutionContext ec) { - switch( getInput1().getDataType() ) { + switch (getInput1().getDataType()) { case FRAME: { FrameBlock fin = ec.getFrameInput(getInput1().getName()); MatrixBlock out = MatrixBlockFromFrame.convertToMatrixBlock(fin, k); @@ -937,31 +860,28 @@ private void processCastAsMatrixVariableInstruction(ExecutionContext ec) { } case LIST: { //TODO handling of cleanup status, potentially new object - ListObject list = (ListObject)ec.getVariable(getInput1().getName()); - if( list.getLength() > 1 ) { - if( !list.checkAllDataTypes(DataType.SCALAR) ) + ListObject list = (ListObject) ec.getVariable(getInput1().getName()); + if (list.getLength() > 1) { + if (!list.checkAllDataTypes(DataType.SCALAR)) throw new DMLRuntimeException("as.matrix over multi-entry list only allows scalars."); MatrixBlock out = new MatrixBlock(list.getLength(), 1, false); - for( int i=0; i dat = colNames.getData(); - for(int i = 0; i < out.getNumColumns();i++) - names[i] = ((StringObject)dat.get(i)).getStringValue(); + for (int i = 0; i < out.getNumColumns(); i++) + names[i] = ((StringObject) dat.get(i)).getStringValue(); out.setColumnNames(names); } } /** * Handler for Read instruction - * + * * @param ec execution context */ - private void processReadInstruction(ExecutionContext ec){ - ec.setScalarOutput(getInput1().getName(), - HDFSTool.readScalarObjectFromHDFSFile(getInput2().getName(), getInput1().getValueType())); + private void processReadInstruction(ExecutionContext ec) { + ec.setScalarOutput(getInput1().getName(), HDFSTool.readScalarObjectFromHDFSFile(getInput2().getName(), getInput1().getValueType())); } /** @@ -1025,15 +942,15 @@ private void processReadInstruction(ExecutionContext ec){ * @param ec execution context */ private void processCopyInstruction(ExecutionContext ec) { - + // get source variable Data dd = ec.getVariable(getInput1().getName()); - if ( dd == null ) - throw new DMLRuntimeException("Unexpected error: could not find a data object for variable name:" + getInput1().getName() + ", while processing instruction " +this.toString()); + if (dd == null) + throw new DMLRuntimeException("Unexpected error: could not find a data object for variable name:" + getInput1().getName() + ", while processing instruction " + this.toString()); if (DMLScript.USE_OOC && dd instanceof MatrixObject) - TeeOOCInstruction.incrRef(((MatrixObject)dd).getStreamable(), 1); + TeeOOCInstruction.incrRef(((MatrixObject) dd).getStreamable(), 1); // remove existing variable bound to target name Data input2_data = ec.removeVariable(getInput2().getName()); @@ -1041,8 +958,7 @@ private void processCopyInstruction(ExecutionContext ec) { TeeOOCInstruction.incrRef(((MatrixObject) input2_data).getStreamable(), -1); //cleanup matrix data on fs/hdfs (if necessary) - if( input2_data != null ) - ec.cleanupDataObject(input2_data); + if (input2_data != null) ec.cleanupDataObject(input2_data); // do the actual copy! ec.setVariable(getInput2().getName(), dd); @@ -1050,7 +966,7 @@ private void processCopyInstruction(ExecutionContext ec) { /** * Handler for write instructions. - * + *

* Non-native formats like MM and CSV are handled through specialized helper functions. * The default behavior is to write out the specified matrix from the instruction, in * the format given by the corresponding symbol table entry. @@ -1062,39 +978,31 @@ private void processWriteInstruction(ExecutionContext ec) { String fname = ec.getScalarInput(getInput2()).getStringValue(); String fmtStr = ec.getScalarInput(getInput3()).getStringValue(); FileFormat fmt = FileFormat.safeValueOf(fmtStr); - if( fmt != FileFormat.LIBSVM && fmt != FileFormat.HDF5) { + if (fmt != FileFormat.LIBSVM && fmt != FileFormat.HDF5) { String desc = ec.getScalarInput(getInput4().getName(), ValueType.STRING, getInput4().isLiteral()).getStringValue(); _formatProperties.setDescription(desc); } - if( getInput1().getDataType() == DataType.SCALAR ) { + if (getInput1().getDataType() == DataType.SCALAR) { HDFSTool.writeScalarToHDFS(ec.getScalarInput(getInput1()), fname); - } - else if( getInput1().getDataType() == DataType.MATRIX ) { - if( fmt == FileFormat.MM ) - writeMMFile(ec, fname); - else if( fmt == FileFormat.CSV ) - writeCSVFile(ec, fname); - else if(fmt == FileFormat.LIBSVM) - writeLIBSVMFile(ec, fname); - else if(fmt == FileFormat.HDF5) - writeHDF5File(ec, fname); + } else if (getInput1().getDataType() == DataType.MATRIX) { + if (fmt == FileFormat.MM) writeMMFile(ec, fname); + else if (fmt == FileFormat.CSV) writeCSVFile(ec, fname); + else if (fmt == FileFormat.LIBSVM) writeLIBSVMFile(ec, fname); + else if (fmt == FileFormat.HDF5) writeHDF5File(ec, fname); else { // Default behavior (text, binary) MatrixObject mo = ec.getMatrixObject(getInput1().getName()); int blen = Integer.parseInt(getInput4().getName()); mo.exportData(fname, fmtStr, new FileFormatProperties(blen)); } - } - else if( getInput1().getDataType() == DataType.FRAME ) { + } else if (getInput1().getDataType() == DataType.FRAME) { FrameObject mo = ec.getFrameObject(getInput1().getName()); mo.exportData(fname, fmtStr, _formatProperties); - } - else if( getInput1().getDataType() == DataType.TENSOR ) { + } else if (getInput1().getDataType() == DataType.TENSOR) { // TODO write tensor TensorObject to = ec.getTensorObject(getInput1().getName()); to.exportData(fname, fmtStr, _formatProperties); - } - else if( getInput1().getDataType() == DataType.LIST ) { + } else if (getInput1().getDataType() == DataType.LIST) { ListObject lo = ec.getListObject(getInput1().getName()); int blen = Integer.parseInt(getInput4().getName()); ListWriter.writeListToHDFS(lo, fname, fmtStr, new FileFormatProperties(blen)); @@ -1103,18 +1011,17 @@ else if( getInput1().getDataType() == DataType.LIST ) { /** * Handler for SetFileName instruction + * * @param ec execution context */ - private void processSetFileNameInstruction(ExecutionContext ec){ + private void processSetFileNameInstruction(ExecutionContext ec) { Data data = ec.getVariable(getInput1().getName()); - if ( data.getDataType() == DataType.MATRIX ) { - if ( getInput3().getName().equalsIgnoreCase("remote") ) - ((MatrixObject)data).setFileName(getInput2().getName()); + if (data.getDataType() == DataType.MATRIX) { + if (getInput3().getName().equalsIgnoreCase("remote")) + ((MatrixObject) data).setFileName(getInput2().getName()); else - throw new DMLRuntimeException( - "Invalid location (" + getInput3().getName() + ") in SetFileName instruction: " + instString); - } - else + throw new DMLRuntimeException("Invalid location (" + getInput3().getName() + ") in SetFileName instruction: " + instString); + } else throw new DMLRuntimeException("Invalid data type (" + getInput1().getDataType() + ") in SetFileName instruction: " + instString); } @@ -1122,51 +1029,45 @@ private void processSetFileNameInstruction(ExecutionContext ec){ * Remove variable instruction externalized as a static function in order to allow various * cleanup procedures to use the same codepath as the actual rmVar instruction * - * @param ec execution context + * @param ec execution context * @param varname variable name */ - public static void processRmvarInstruction( ExecutionContext ec, String varname ) { + public static void processRmvarInstruction(ExecutionContext ec, String varname) { // remove variable from symbol table Data dat = ec.removeVariable(varname); if (DMLScript.USE_OOC && dat instanceof MatrixObject) TeeOOCInstruction.incrRef(((MatrixObject) dat).getStreamable(), -1); //cleanup matrix data on fs/hdfs (if necessary) - if( dat != null ) - ec.cleanupDataObject(dat); + if (dat != null) ec.cleanupDataObject(dat); } /** * Helper function to write CSV files to HDFS. * - * @param ec execution context + * @param ec execution context * @param fname file name */ private void writeCSVFile(ExecutionContext ec, String fname) { MatrixObject mo = ec.getMatrixObject(getInput1().getName()); String outFmt = "csv"; - FileFormatProperties fprop = (_formatProperties instanceof FileFormatPropertiesCSV) ? - _formatProperties : new FileFormatPropertiesCSV(); //for dynamic format strings - - if(mo.isDirty()) { + FileFormatProperties fprop = (_formatProperties instanceof FileFormatPropertiesCSV) ? _formatProperties : new FileFormatPropertiesCSV(); //for dynamic format strings + + if (mo.isDirty()) { // there exist data computed in CP that is not backed up on HDFS // i.e., it is either in-memory or in evicted space mo.exportData(fname, outFmt, fprop); - } - else { + } else { try { - FileFormat fmt = ((MetaDataFormat)mo.getMetaData()).getFileFormat(); + FileFormat fmt = ((MetaDataFormat) mo.getMetaData()).getFileFormat(); DataCharacteristics dc = (mo.getMetaData()).getDataCharacteristics(); - if( fmt == FileFormat.CSV && !mo.isPersistentRead() ) { - WriterTextCSV writer = new WriterTextCSV((FileFormatPropertiesCSV)fprop); + if (fmt == FileFormat.CSV && !mo.isPersistentRead()) { + WriterTextCSV writer = new WriterTextCSV((FileFormatPropertiesCSV) fprop); writer.addHeaderToCSV(mo.getFileName(), fname, dc.getRows(), dc.getCols()); - } - else { + } else { mo.exportData(fname, outFmt, fprop); } - HDFSTool.writeMetaDataFile(fname + ".mtd", - mo.getValueType(), dc, FileFormat.CSV, fprop); - } - catch(IOException e) { + HDFSTool.writeMetaDataFile(fname + ".mtd", mo.getValueType(), dc, FileFormat.CSV, fprop); + } catch (IOException e) { throw new DMLRuntimeException(e); } } @@ -1175,25 +1076,22 @@ private void writeCSVFile(ExecutionContext ec, String fname) { /** * Helper function to write LIBSVM files to HDFS. * - * @param ec execution context + * @param ec execution context * @param fname file name */ private void writeLIBSVMFile(ExecutionContext ec, String fname) { MatrixObject mo = ec.getMatrixObject(getInput1().getName()); String outFmt = "libsvm"; - if(mo.isDirty()) { + if (mo.isDirty()) { // there exist data computed in CP that is not backed up on HDFS // i.e., it is either in-memory or in evicted space mo.exportData(fname, outFmt, _formatProperties); - } - else { + } else { try { mo.exportData(fname, outFmt, _formatProperties); - HDFSTool.writeMetaDataFile(fname + ".mtd", mo.getValueType(), - mo.getMetaData().getDataCharacteristics(), FileFormat.LIBSVM, _formatProperties); - } - catch (IOException e) { + HDFSTool.writeMetaDataFile(fname + ".mtd", mo.getValueType(), mo.getMetaData().getDataCharacteristics(), FileFormat.LIBSVM, _formatProperties); + } catch (IOException e) { throw new DMLRuntimeException(e); } } @@ -1209,26 +1107,22 @@ private void writeHDF5File(ExecutionContext ec, String fname) { MatrixObject mo = ec.getMatrixObject(getInput1().getName()); String outFmt = "hdf5"; - if(mo.isDirty()) { + if (mo.isDirty()) { // there exist data computed in CP that is not backed up on HDFS // i.e., it is either in-memory or in evicted space mo.exportData(fname, outFmt, _formatProperties); - } - else { + } else { try { FileFormat fmt = ((MetaDataFormat) mo.getMetaData()).getFileFormat(); DataCharacteristics dc = (mo.getMetaData()).getDataCharacteristics(); - if(fmt == FileFormat.HDF5 && !getInput1().getName().startsWith(org.apache.sysds.lops.Data.PREAD_PREFIX)) { + if (fmt == FileFormat.HDF5 && !getInput1().getName().startsWith(org.apache.sysds.lops.Data.PREAD_PREFIX)) { //FIXME why is this writer never used? - @SuppressWarnings("unused") - WriterHDF5 writer = new WriterHDF5((FileFormatPropertiesHDF5) _formatProperties); - } - else { + @SuppressWarnings("unused") WriterHDF5 writer = new WriterHDF5((FileFormatPropertiesHDF5) _formatProperties); + } else { mo.exportData(fname, outFmt, _formatProperties); } HDFSTool.writeMetaDataFile(fname + ".mtd", mo.getValueType(), dc, FileFormat.HDF5, _formatProperties); - } - catch (IOException e) { + } catch (IOException e) { throw new DMLRuntimeException(e); } } @@ -1237,32 +1131,26 @@ private void writeHDF5File(ExecutionContext ec, String fname) { /** * Helper function to write MM files to HDFS. * - * @param ec execution context + * @param ec execution context * @param fname file name */ private void writeMMFile(ExecutionContext ec, String fname) { MatrixObject mo = ec.getMatrixObject(getInput1().getName()); String outFmt = FileFormat.MM.toString(); - if(mo.isDirty()) { + if (mo.isDirty()) { // there exist data computed in CP that is not backed up on HDFS // i.e., it is either in-memory or in evicted space mo.exportData(fname, outFmt); - } - else { + } else { try { - FileFormat fmt = ((MetaDataFormat)mo.getMetaData()).getFileFormat(); + FileFormat fmt = ((MetaDataFormat) mo.getMetaData()).getFileFormat(); DataCharacteristics dc = mo.getDataCharacteristics(); - if( fmt == FileFormat.TEXT - && !getInput1().getName().startsWith(org.apache.sysds.lops.Data.PREAD_PREFIX) ) - { - WriterMatrixMarket.mergeTextcellToMatrixMarket(mo.getFileName(), - fname, dc.getRows(), dc.getCols(), dc.getNonZeros()); - } - else { + if (fmt == FileFormat.TEXT && !getInput1().getName().startsWith(org.apache.sysds.lops.Data.PREAD_PREFIX)) { + WriterMatrixMarket.mergeTextcellToMatrixMarket(mo.getFileName(), fname, dc.getRows(), dc.getCols(), dc.getNonZeros()); + } else { mo.exportData(fname, outFmt); } - } - catch (IOException e) { + } catch (IOException e) { throw new DMLRuntimeException(e); } } @@ -1291,7 +1179,7 @@ public static Instruction prepareRemoveInstruction(String... varNames) { sb.append("CP"); sb.append(Lop.OPERAND_DELIMITOR); sb.append(Opcodes.RMVAR); - for( String varName : varNames ) { + for (String varName : varNames) { sb.append(Lop.OPERAND_DELIMITOR); sb.append(varName); } @@ -1299,30 +1187,25 @@ public static Instruction prepareRemoveInstruction(String... varNames) { } public static Instruction prepareCopyInstruction(String srcVar, String destVar) { - return parseInstruction( - InstructionUtils.concatOperands("CP", Opcodes.CPVAR.toString(), srcVar, destVar)); + return parseInstruction(InstructionUtils.concatOperands("CP", Opcodes.CPVAR.toString(), srcVar, destVar)); } public static Instruction prepMoveInstruction(String srcVar, String destFileName, String format) { - return parseInstruction( - InstructionUtils.concatOperands("CP", Opcodes.MVVAR.toString(), srcVar, destFileName, format)); + return parseInstruction(InstructionUtils.concatOperands("CP", Opcodes.MVVAR.toString(), srcVar, destFileName, format)); } public static Instruction prepMoveInstruction(String srcVar, String destVar) { - return parseInstruction( - InstructionUtils.concatOperands("CP", Opcodes.MVVAR.toString(), srcVar, destVar)); + return parseInstruction(InstructionUtils.concatOperands("CP", Opcodes.MVVAR.toString(), srcVar, destVar)); } private static String getBasicCreatevarString(String varName, String fileName, boolean fNameOverride, DataType dt, String format) { //note: the filename override property leads to concatenation of unique ids in order to //ensure conflicting filenames for objects that originate from the same instruction - boolean lfNameOverride = fNameOverride && !ConfigurationManager - .getCompilerConfigFlag(ConfigType.IGNORE_TEMPORARY_FILENAMES); + boolean lfNameOverride = fNameOverride && !ConfigurationManager.getCompilerConfigFlag(ConfigType.IGNORE_TEMPORARY_FILENAMES); // Constant CREATEVAR_FILE_NAME_VAR_POS is used to find a position of filename within a string generated through this function. // If this position of filename within this string changes then constant CREATEVAR_FILE_NAME_VAR_POS to be updated. - return InstructionUtils.concatOperands( - "CP", Opcodes.CREATEVAR.toString(), varName, fileName, String.valueOf(lfNameOverride), dt.toString(), format); + return InstructionUtils.concatOperands("CP", Opcodes.CREATEVAR.toString(), varName, fileName, String.valueOf(lfNameOverride), dt.toString(), format); } public static Instruction prepCreatevarInstruction(String varName, String fileName, boolean fNameOverride, String format) { @@ -1330,56 +1213,45 @@ public static Instruction prepCreatevarInstruction(String varName, String fileNa } public static Instruction prepCreatevarInstruction(String varName, String fileName, boolean fNameOverride, DataType dt, String format, DataCharacteristics mc, UpdateType update) { - return parseInstruction(InstructionUtils.concatOperands( - getBasicCreatevarString(varName, fileName, fNameOverride, dt, format), - String.valueOf(mc.getRows()), String.valueOf(mc.getCols()), String.valueOf(mc.getBlocksize()), - String.valueOf(mc.getNonZeros()), update.toString().toLowerCase())); + return parseInstruction(InstructionUtils.concatOperands(getBasicCreatevarString(varName, fileName, fNameOverride, dt, format), String.valueOf(mc.getRows()), String.valueOf(mc.getCols()), String.valueOf(mc.getBlocksize()), String.valueOf(mc.getNonZeros()), update.toString().toLowerCase())); } public static Instruction prepCreatevarInstruction(String varName, String fileName, boolean fNameOverride, DataType dt, String format, DataCharacteristics mc, UpdateType update, boolean hasHeader, String delim, boolean sparse) { - return parseInstruction(InstructionUtils.concatOperands( - getBasicCreatevarString(varName, fileName, fNameOverride, dt, format), - String.valueOf(mc.getRows()), String.valueOf(mc.getCols()), String.valueOf(mc.getBlocksize()), - String.valueOf(mc.getNonZeros()), update.toString().toLowerCase(), - String.valueOf(hasHeader), delim, String.valueOf(sparse))); + return parseInstruction(InstructionUtils.concatOperands(getBasicCreatevarString(varName, fileName, fNameOverride, dt, format), String.valueOf(mc.getRows()), String.valueOf(mc.getCols()), String.valueOf(mc.getBlocksize()), String.valueOf(mc.getNonZeros()), update.toString().toLowerCase(), String.valueOf(hasHeader), delim, String.valueOf(sparse))); } @Override public void updateInstructionThreadID(String pattern, String replace) { - if( opcode == VariableOperationCode.CreateVariable - || opcode == VariableOperationCode.SetFileName ) - { + if (opcode == VariableOperationCode.CreateVariable || opcode == VariableOperationCode.SetFileName) { //replace in-memory instruction getInput2().setName(getInput2().getName().replaceAll(pattern, replace)); // Find a start position of file name string. int iPos = StringUtils.ordinalIndexOf(instString, Lop.OPERAND_DELIMITOR, CREATEVAR_FILE_NAME_VAR_POS); // Find an end position of file name string. - int iPos2 = StringUtils.indexOf(instString, Lop.OPERAND_DELIMITOR, iPos+1); + int iPos2 = StringUtils.indexOf(instString, Lop.OPERAND_DELIMITOR, iPos + 1); StringBuilder sb = new StringBuilder(); - sb.append(instString.substring(0,iPos+1)); // It takes first part before file name. + sb.append(instString.substring(0, iPos + 1)); // It takes first part before file name. // This will replace 'pattern' with 'replace' string from file name. - sb.append(ProgramConverter.saveReplaceFilenameThreadID(instString.substring(iPos+1, iPos2+1), pattern, replace)); - sb.append(instString.substring(iPos2+1)); // It takes last part after file name. + sb.append(ProgramConverter.saveReplaceFilenameThreadID(instString.substring(iPos + 1, iPos2 + 1), pattern, replace)); + sb.append(instString.substring(iPos2 + 1)); // It takes last part after file name. instString = sb.toString(); } } @Override - public Pair getLineageItem(ExecutionContext ec) { + public Pair getLineageItem(ExecutionContext ec) { String varname = null; LineageItem li = null; switch (getVariableOpcode()) { case CreateVariable: - if (!_containsPreadPrefix) - break; //otherwise fall through + if (!_containsPreadPrefix) break; //otherwise fall through case Read: { varname = getInput1().getName(); - li = new LineageItem(toString().replace(getInput1().getName(), - org.apache.sysds.lops.Data.PREAD_PREFIX+"xxx"), getOpcode()); + li = new LineageItem(toString().replace(getInput1().getName(), org.apache.sysds.lops.Data.PREAD_PREFIX + "xxx"), getOpcode()); break; } case AssignVariable: { @@ -1397,8 +1269,7 @@ public Pair getLineageItem(ExecutionContext ec) { case Write: { ArrayList lineages = new ArrayList<>(); for (CPOperand input : getInputs()) - if (!input.getName().isEmpty()) - lineages.add(ec.getLineage().getOrCreate(input)); + if (!input.getName().isEmpty()) lineages.add(ec.getLineage().getOrCreate(input)); if (_formatProperties != null && _formatProperties.getDescription() != null && !_formatProperties.getDescription().isEmpty()) lineages.add(new LineageItem(_formatProperties.getDescription())); varname = getInput1().getName(); @@ -1410,7 +1281,7 @@ public Pair getLineageItem(ExecutionContext ec) { case CastAsIntegerVariable: case CastAsScalarVariable: case CastAsMatrixVariable: - case CastAsFrameVariable:{ + case CastAsFrameVariable: { varname = getOutputVariableName(); li = new LineageItem(getOpcode(), LineageItemUtils.getLineage(ec, getInput1())); break; @@ -1420,8 +1291,7 @@ public Pair getLineageItem(ExecutionContext ec) { ListObject lobj = ec.getListObject(getInput1()); if (lobj.getLength() != 1 || !(lobj.getData(0) instanceof ListObject)) li = new LineageItem(getOpcode(), LineageItemUtils.getLineage(ec, getInput1())); - else - li = new LineageItem(getOpcode(), new LineageItem[] {lobj.getLineageItem(0)}); + else li = new LineageItem(getOpcode(), new LineageItem[]{lobj.getLineageItem(0)}); break; case RemoveVariable: case MoveVariable: @@ -1432,12 +1302,7 @@ public Pair getLineageItem(ExecutionContext ec) { } public boolean isVariableCastInstruction() { - return opcode == VariableOperationCode.CastAsScalarVariable - || opcode == VariableOperationCode.CastAsMatrixVariable - || opcode == VariableOperationCode.CastAsFrameVariable - || opcode == VariableOperationCode.CastAsIntegerVariable - || opcode == VariableOperationCode.CastAsDoubleVariable - || opcode == VariableOperationCode.CastAsBooleanVariable; + return opcode == VariableOperationCode.CastAsScalarVariable || opcode == VariableOperationCode.CastAsMatrixVariable || opcode == VariableOperationCode.CastAsFrameVariable || opcode == VariableOperationCode.CastAsIntegerVariable || opcode == VariableOperationCode.CastAsDoubleVariable || opcode == VariableOperationCode.CastAsBooleanVariable; } public static String getUniqueFileName(String fname) { diff --git a/src/main/java/org/apache/sysds/runtime/instructions/fed/ParameterizedBuiltinFEDInstruction.java b/src/main/java/org/apache/sysds/runtime/instructions/fed/ParameterizedBuiltinFEDInstruction.java index f19b51e2985..48c53efa937 100644 --- a/src/main/java/org/apache/sysds/runtime/instructions/fed/ParameterizedBuiltinFEDInstruction.java +++ b/src/main/java/org/apache/sysds/runtime/instructions/fed/ParameterizedBuiltinFEDInstruction.java @@ -841,14 +841,22 @@ public FederatedResponse execute(ExecutionContext ec, Data... data) { String[] colNames = _meta.getColumnNames(); FrameBlock fbout = _decoder.decode(mb, new FrameBlock(_decoder.getSchema())); - fbout.setColumnNames(Arrays.copyOfRange(colNames, 0, fbout.getNumColumns())); + + fbout.setColumnNames(Arrays.copyOfRange(colNames,0, fbout.getNumColumns())); // copy characteristics MatrixCharacteristics mc = new MatrixCharacteristics(mo.getDataCharacteristics()); + FrameObject fo = new FrameObject(OptimizerUtils.getUniqueTempFileName(), new MetaDataFormat(mc, Types.FileFormat.BINARY)); + // set the encoded data fo.acquireModify(fbout); + + // set schema and column names + fo.setSchema(fbout.getSchema()); + fo.setColumnNames(Arrays.copyOfRange(colNames,0, fbout.getNumColumns())); + fo.release(); mo.release(); diff --git a/src/main/java/org/apache/sysds/runtime/instructions/spark/BinaryFrameFrameSPInstruction.java b/src/main/java/org/apache/sysds/runtime/instructions/spark/BinaryFrameFrameSPInstruction.java index 9c9f9dcfd82..979e1b87819 100644 --- a/src/main/java/org/apache/sysds/runtime/instructions/spark/BinaryFrameFrameSPInstruction.java +++ b/src/main/java/org/apache/sysds/runtime/instructions/spark/BinaryFrameFrameSPInstruction.java @@ -65,6 +65,11 @@ else if(getOpcode().equals(Opcodes.APPLYSCHEMA.toString())){ out = in1.mapValues(new applySchema(fb.getValue())); sec.releaseFrameInput(input2.getName()); } + else if(getOpcode().equals(Opcodes.SET_COLNAMES.toString())) { + Broadcast fb = sec.getSparkContext().broadcast(sec.getFrameInput(input2.getName())); + out = in1.mapValues(new setColumnNames(fb.getValue())); + sec.releaseFrameInput(input2.getName()); + } else { JavaPairRDD in2 = sec.getFrameBinaryBlockRDDHandleForVariable(input2.getName()); // create output frame @@ -140,4 +145,22 @@ public FrameBlock call(FrameBlock arg0) throws Exception { return arg0.applySchema(schema); } } + + private static class setColumnNames implements Function{ + //private static final long serialVersionUID = 1L; + + private String[] columnNames; + + public setColumnNames(FrameBlock names) { + columnNames = new String[names.getNumColumns()]; + for(int i = 0; i < columnNames.length; i++) + columnNames[i] = names.get(0, i).toString(); + } + + @Override + public FrameBlock call(FrameBlock arg0) throws Exception { + arg0.setColumnNames(columnNames); + return arg0; + } + } } \ No newline at end of file diff --git a/src/main/java/org/apache/sysds/runtime/instructions/spark/BuiltinNarySPInstruction.java b/src/main/java/org/apache/sysds/runtime/instructions/spark/BuiltinNarySPInstruction.java index ea59fe99e2b..0e380fe9205 100644 --- a/src/main/java/org/apache/sysds/runtime/instructions/spark/BuiltinNarySPInstruction.java +++ b/src/main/java/org/apache/sysds/runtime/instructions/spark/BuiltinNarySPInstruction.java @@ -43,6 +43,7 @@ import org.apache.sysds.runtime.instructions.spark.AppendGSPInstruction.ShiftMatrix; import org.apache.sysds.runtime.instructions.spark.functions.MapInputSignature; import org.apache.sysds.runtime.instructions.spark.functions.MapJoinSignature; +import org.apache.sysds.runtime.instructions.spark.functions.SetColumnNamesFunction; import org.apache.sysds.runtime.instructions.spark.utils.RDDAggregateUtils; import org.apache.sysds.runtime.instructions.spark.utils.SparkUtils; import org.apache.sysds.runtime.lineage.LineageItem; @@ -66,44 +67,43 @@ import static org.apache.sysds.runtime.instructions.spark.FrameAppendMSPInstruction.appendFrameMSP; import static org.apache.sysds.runtime.instructions.spark.FrameAppendRSPInstruction.appendFrameRSP; -public class BuiltinNarySPInstruction extends SPInstruction implements LineageTraceable -{ +public class BuiltinNarySPInstruction extends SPInstruction implements LineageTraceable { public CPOperand[] inputs; public CPOperand output; - + protected BuiltinNarySPInstruction(CPOperand[] in, CPOperand out, String opcode, String istr) { super(SPType.BuiltinNary, opcode, istr); inputs = in; output = out; } - public static BuiltinNarySPInstruction parseInstruction ( String str ) { + public static BuiltinNarySPInstruction parseInstruction(String str) { String[] parts = InstructionUtils.getInstructionPartsWithValueType(str); String opcode = parts[0]; CPOperand output = new CPOperand(parts[parts.length - 1]); CPOperand[] inputs = null; inputs = new CPOperand[parts.length - 2]; - for (int i = 1; i < parts.length-1; i++) - inputs[i-1] = new CPOperand(parts[i]); + for (int i = 1; i < parts.length - 1; i++) + inputs[i - 1] = new CPOperand(parts[i]); return new BuiltinNarySPInstruction(inputs, output, opcode, str); } - @Override + @Override public void processInstruction(ExecutionContext ec) { - SparkExecutionContext sec = (SparkExecutionContext)ec; - JavaPairRDD out = null; + SparkExecutionContext sec = (SparkExecutionContext) ec; + JavaPairRDD out = null; DataCharacteristics dcout = null; boolean inputIsMatrix = inputs[0].isMatrix(); - - if( getOpcode().equals(Opcodes.CBIND.toString()) || getOpcode().equals(Opcodes.RBIND.toString()) ) { + + if (getOpcode().equals(Opcodes.CBIND.toString()) || getOpcode().equals(Opcodes.RBIND.toString())) { //compute output characteristics boolean cbind = getOpcode().equals(Opcodes.CBIND.toString()); dcout = computeAppendOutputDataCharacteristics(sec, inputs, cbind); - if(inputIsMatrix){ + if (inputIsMatrix) { //get consolidated input via union over shifted and padded inputs DataCharacteristics off = new MatrixCharacteristics(0, 0, dcout.getBlocksize(), 0); - for( CPOperand input : inputs ) { + for (CPOperand input : inputs) { DataCharacteristics mcIn = sec.getDataCharacteristics(input.getName()); JavaPairRDD in = sec .getBinaryMatrixBlockRDDHandleForVariable(input.getName()) @@ -118,77 +118,83 @@ public void processInstruction(ExecutionContext ec) { } //FRAME else { - JavaPairRDD outFrame = - sec.getFrameBinaryBlockRDDHandleForVariable( inputs[0].getName() ); + JavaPairRDD outFrame = + sec.getFrameBinaryBlockRDDHandleForVariable(inputs[0].getName()); dcout = new MatrixCharacteristics(sec.getDataCharacteristics(inputs[0].getName())); FrameObject fo = new FrameObject(sec.getFrameObject(inputs[0].getName())); boolean[] broadcasted = new boolean[inputs.length]; broadcasted[0] = false; - for(int i = 1; i < inputs.length; i++){ + for (int i = 1; i < inputs.length; i++) { DataCharacteristics dcIn = sec.getDataCharacteristics(inputs[i].getName()); final int blk_size = dcout.getBlocksize() <= 0 ? DEFAULT_FRAME_BLOCKSIZE : dcout.getBlocksize(); broadcasted[i] = BinaryOp.FORCED_APPEND_METHOD == MR_MAPPEND - || BinaryOp.FORCED_APPEND_METHOD == null && cbind && dcIn.getCols() <= blk_size + || BinaryOp.FORCED_APPEND_METHOD == null && cbind && dcIn.getCols() <= blk_size && OptimizerUtils.checkSparkBroadcastMemoryBudget( - dcout.getCols(), dcIn.getCols(), blk_size, dcIn.getNonZeros()); + dcout.getCols(), dcIn.getCols(), blk_size, dcIn.getNonZeros()); //easy case: broadcast & map - if(broadcasted[i]){ + if (broadcasted[i]) { outFrame = appendFrameMSP(outFrame, sec.getBroadcastForFrameVariable(inputs[i].getName())); } //general case for frames: - else{ - if(BinaryOp.FORCED_APPEND_METHOD != null && BinaryOp.FORCED_APPEND_METHOD != MR_RAPPEND) + else { + if (BinaryOp.FORCED_APPEND_METHOD != null && BinaryOp.FORCED_APPEND_METHOD != MR_RAPPEND) throw new DMLRuntimeException("Forced append type [" - +BinaryOp.FORCED_APPEND_METHOD+"] is not supported for frames"); + + BinaryOp.FORCED_APPEND_METHOD + "] is not supported for frames"); - JavaPairRDD in2 = - sec.getFrameBinaryBlockRDDHandleForVariable(inputs[i].getName() ); + JavaPairRDD in2 = + sec.getFrameBinaryBlockRDDHandleForVariable(inputs[i].getName()); outFrame = appendFrameRSP(outFrame, in2, dcout.getRows(), cbind); } updateAppendDataCharacteristics(dcIn, dcout, cbind); - if(cbind) + if (cbind) { fo.setSchema(fo.mergeSchemas(sec.getFrameObject(inputs[i].getName()))); + String[] outputNames = ArrayUtils.addAll(fo.getColumnNames(), + sec.getFrameObject(inputs[i].getName()).getColumnNames()); + fo.setColumnNames(outputNames); + } } //set output RDD and add lineage sec.getDataCharacteristics(output.getName()).set(dcout); + outFrame = outFrame.mapValues(new SetColumnNamesFunction(fo.getColumnNames())); sec.setRDDHandleForVariable(output.getName(), outFrame); - sec.getFrameObject(output.getName()).setSchema(fo.getSchema()); - for( int i = 0; i < inputs.length; i++) - if(broadcasted[i]) + FrameObject outputFrame = sec.getFrameObject(output.getName()); + outputFrame.setSchema(fo.getSchema()); + outputFrame.setColumnNames(fo.getColumnNames()); + for (int i = 0; i < inputs.length; i++) + if (broadcasted[i]) sec.addLineageBroadcast(output.getName(), inputs[i].getName()); else sec.addLineageRDD(output.getName(), inputs[i].getName()); return; } - } - else if( ArrayUtils.contains(new String[]{Opcodes.NMIN.toString(),Opcodes.NMAX.toString(),Opcodes.NP.toString(),Opcodes.NM.toString()}, getOpcode()) ) { + } else if (ArrayUtils.contains(new String[]{Opcodes.NMIN.toString(), Opcodes.NMAX.toString(), Opcodes.NP.toString(), Opcodes.NM.toString()}, getOpcode())) { //compute output characteristics dcout = computeMinMaxOutputDataCharacteristics(sec, inputs); - + //get scalars and consolidated input via join List scalars = sec.getScalarInputs(inputs); JavaPairRDD in = null; - for( CPOperand input : inputs ) { - if( !input.getDataType().isMatrix() ) continue; + for (CPOperand input : inputs) { + if (!input.getDataType().isMatrix()) continue; JavaPairRDD tmp = sec - .getBinaryMatrixBlockRDDHandleForVariable(input.getName()); + .getBinaryMatrixBlockRDDHandleForVariable(input.getName()); in = (in == null) ? tmp.mapValues(new MapInputSignature()) : - in.join(tmp).mapValues(new MapJoinSignature()); + in.join(tmp).mapValues(new MapJoinSignature()); } - + //compute nary min/max (partitioning-preserving) out = in.mapValues(new MinMaxAddMultFunction(getOpcode(), scalars)); } - + //set output RDD and add lineage sec.getDataCharacteristics(output.getName()).set(dcout); sec.setRDDHandleForVariable(output.getName(), out); - for( CPOperand input : inputs ) - if( !input.isScalar() ) + for (CPOperand input : inputs) + if (!input.isScalar()) sec.addLineageRDD(output.getName(), input.getName()); } @@ -207,10 +213,10 @@ public Iterator> call(Tuple2 longFram FrameBlock fb = longFrameBlockTuple2._2; ArrayList> list = new ArrayList>(); //single output block - if(max_rows <= DEFAULT_FRAME_BLOCKSIZE){ + if (max_rows <= DEFAULT_FRAME_BLOCKSIZE) { FrameBlock fbout = new FrameBlock(fb.getSchema()); fbout.ensureAllocatedColumns((int) max_rows); - fbout = fbout.leftIndexingOperations(fb,index.intValue() - 1, index.intValue() + fb.getNumRows() - 2,0, fb.getNumColumns()-1, null ); + fbout = fbout.leftIndexingOperations(fb, index.intValue() - 1, index.intValue() + fb.getNumRows() - 2, 0, fb.getNumColumns() - 1, null); list.add(new Tuple2<>(1L, fbout)); } else { throw new NotImplementedException("Other Alignment strategies need to be implemented"); @@ -225,23 +231,23 @@ public Iterator> call(Tuple2 longFram private static DataCharacteristics computeAppendOutputDataCharacteristics(SparkExecutionContext sec, CPOperand[] inputs, boolean cbind) { DataCharacteristics mcIn1 = sec.getDataCharacteristics(inputs[0].getName()); DataCharacteristics mcOut = new MatrixCharacteristics(0, 0, mcIn1.getBlocksize(), 0); - for( CPOperand input : inputs ) { + for (CPOperand input : inputs) { DataCharacteristics mcIn = sec.getDataCharacteristics(input.getName()); updateAppendDataCharacteristics(mcIn, mcOut, cbind); } return mcOut; } - + private static void updateAppendDataCharacteristics(DataCharacteristics in, DataCharacteristics out, boolean cbind) { - out.setDimension(cbind ? Math.max(out.getRows(), in.getRows()) : out.getRows()+in.getRows(), - cbind ? out.getCols()+in.getCols() : Math.max(out.getCols(), in.getCols())); - out.setNonZeros((out.getNonZeros()!=-1 && in.dimsKnown(true)) ? out.getNonZeros()+in.getNonZeros() : -1); + out.setDimension(cbind ? Math.max(out.getRows(), in.getRows()) : out.getRows() + in.getRows(), + cbind ? out.getCols() + in.getCols() : Math.max(out.getCols(), in.getCols())); + out.setNonZeros((out.getNonZeros() != -1 && in.dimsKnown(true)) ? out.getNonZeros() + in.getNonZeros() : -1); } - + private static DataCharacteristics computeMinMaxOutputDataCharacteristics(SparkExecutionContext sec, CPOperand[] inputs) { DataCharacteristics mcOut = new MatrixCharacteristics(); - for( CPOperand input : inputs ) { - if( !input.getDataType().isMatrix() ) continue; + for (CPOperand input : inputs) { + if (!input.getDataType().isMatrix()) continue; DataCharacteristics mcIn = sec.getDataCharacteristics(input.getName()); mcOut.setRows(Math.max(mcOut.getRows(), mcIn.getRows())); mcOut.setCols(Math.max(mcOut.getCols(), mcIn.getCols())); @@ -249,13 +255,12 @@ private static DataCharacteristics computeMinMaxOutputDataCharacteristics(SparkE } return mcOut; } - - public static class PadBlocksFunction implements PairFunction,MatrixIndexes,MatrixBlock> - { + + public static class PadBlocksFunction implements PairFunction, MatrixIndexes, MatrixBlock> { private static final long serialVersionUID = 1291358959908299855L; - + private final DataCharacteristics _mcOut; - + public PadBlocksFunction(DataCharacteristics mcOut) { _mcOut = mcOut; } @@ -266,23 +271,23 @@ public Tuple2 call(Tuple2 mb.getNumRows() ) //rbind - mb = mb.append(new MatrixBlock(brlen-mb.getNumRows(),bclen,true), new MatrixBlock(), false); - else if( bclen > mb.getNumColumns() ) //cbind - mb = mb.append(new MatrixBlock(brlen,bclen-mb.getNumColumns(),true), new MatrixBlock(), true); + if (brlen > mb.getNumRows()) //rbind + mb = mb.append(new MatrixBlock(brlen - mb.getNumRows(), bclen, true), new MatrixBlock(), false); + else if (bclen > mb.getNumColumns()) //cbind + mb = mb.append(new MatrixBlock(brlen, bclen - mb.getNumColumns(), true), new MatrixBlock(), true); return new Tuple2<>(ix, mb); } } - + private static class MinMaxAddMultFunction implements Function { private static final long serialVersionUID = -4227447915387484397L; - + private final SimpleOperator _op; private final ScalarObject[] _scalars; @@ -292,16 +297,16 @@ public MinMaxAddMultFunction(String opcode, List scalars) { opcode.equals(Opcodes.NM.toString()) ? Multiply.getMultiplyFnObject() : Builtin.getBuiltinFnObject(opcode.substring(1))); } - + @Override public MatrixBlock call(MatrixBlock[] v1) throws Exception { return MatrixBlock.naryOperations(_op, v1, _scalars, new MatrixBlock()); } } - + @Override public Pair getLineageItem(ExecutionContext ec) { return Pair.of(output.getName(), new LineageItem(getOpcode(), - LineageItemUtils.getLineage(ec, inputs))); + LineageItemUtils.getLineage(ec, inputs))); } } diff --git a/src/main/java/org/apache/sysds/runtime/instructions/spark/CSVReblockSPInstruction.java b/src/main/java/org/apache/sysds/runtime/instructions/spark/CSVReblockSPInstruction.java index 920e7764df9..18527695eed 100644 --- a/src/main/java/org/apache/sysds/runtime/instructions/spark/CSVReblockSPInstruction.java +++ b/src/main/java/org/apache/sysds/runtime/instructions/spark/CSVReblockSPInstruction.java @@ -49,7 +49,7 @@ import org.apache.sysds.utils.Statistics; public class CSVReblockSPInstruction extends UnarySPInstruction { - + private int _blen; private boolean _hasHeader; private String _delim; @@ -58,7 +58,7 @@ public class CSVReblockSPInstruction extends UnarySPInstruction { private Set _naStrings; protected CSVReblockSPInstruction(Operator op, CPOperand in, CPOperand out, int br, int bc, boolean hasHeader, - String delim, boolean fill, double fillValue, String opcode, String instr, Set naStrings) { + String delim, boolean fill, double fillValue, String opcode, String instr, Set naStrings) { super(SPType.CSVReblock, op, in, out, opcode, instr); _blen = br; _blen = bc; @@ -71,7 +71,7 @@ protected CSVReblockSPInstruction(Operator op, CPOperand in, CPOperand out, int public static CSVReblockSPInstruction parseInstruction(String str) { String opcode = InstructionUtils.getOpCode(str); - if( !opcode.equals(Opcodes.CSVRBLK.toString()) ) + if (!opcode.equals(Opcodes.CSVRBLK.toString())) throw new DMLRuntimeException("Incorrect opcode for CSVReblockSPInstruction:" + opcode); // Example parts of CSVReblockSPInstruction: @@ -90,14 +90,14 @@ public static CSVReblockSPInstruction parseInstruction(String str) { String[] naS = parts[8].split(DataExpression.DELIM_NA_STRING_SEP); - if(naS.length > 0 && !(naS.length ==1 && naS[0].isEmpty())){ + if (naS.length > 0 && !(naS.length == 1 && naS[0].isEmpty())) { naStrings = new HashSet<>(); - for(String s: naS) + for (String s : naS) naStrings.add(s); } return new CSVReblockSPInstruction(null, in, out, blen, blen, - hasHeader, delim, fill, fillValue, opcode, str, naStrings); + hasHeader, delim, fill, fillValue, opcode, str, naStrings); } @Override @@ -111,55 +111,61 @@ public void processInstruction(ExecutionContext ec) { throw new DMLRuntimeException("The given format is not implemented for " + "CSVReblockSPInstruction:" + iimd.getFileFormat().toString()); } - + //set output characteristics DataCharacteristics mcIn = sec.getDataCharacteristics(input1.getName()); DataCharacteristics mcOut = sec.getDataCharacteristics(output.getName()); mcOut.set(mcIn.getRows(), mcIn.getCols(), _blen); + if (input1.getDataType() == DataType.FRAME) { + FrameObject inputFrame = sec.getFrameObject(input1.getName()); + FrameObject outputFrame = sec.getFrameObject(output.getName()); + outputFrame.setColumnNames(inputFrame.getColumnNames()); + } + //check for in-memory reblock (w/ lazy spark context, potential for latency reduction) - if( Recompiler.checkCPReblock(sec, input1.getName()) ) { - if( input1.getDataType().isMatrix() || input1.getDataType().isFrame() ) { + if (Recompiler.checkCPReblock(sec, input1.getName())) { + if (input1.getDataType().isMatrix() || input1.getDataType().isFrame()) { Recompiler.executeInMemoryReblock(sec, input1.getName(), output.getName()); } Statistics.decrementNoOfExecutedSPInst(); return; } - + //execute matrix/frame csvreblock - JavaPairRDD out = null; - if( input1.getDataType() == DataType.MATRIX ) + JavaPairRDD out = null; + if (input1.getDataType() == DataType.MATRIX) out = processMatrixCSVReblockInstruction(sec, mcOut); - else if( input1.getDataType() == DataType.FRAME ) - out = processFrameCSVReblockInstruction(sec, mcOut, ((FrameObject)obj).getSchema()); - + else if (input1.getDataType() == DataType.FRAME) + out = processFrameCSVReblockInstruction(sec, mcOut, ((FrameObject) obj).getSchema()); + // put output RDD handle into symbol table sec.setRDDHandleForVariable(output.getName(), out); sec.addLineageRDD(output.getName(), input1.getName()); } @SuppressWarnings("unchecked") - protected JavaPairRDD processMatrixCSVReblockInstruction(SparkExecutionContext sec, DataCharacteristics mcOut) { + protected JavaPairRDD processMatrixCSVReblockInstruction(SparkExecutionContext sec, DataCharacteristics mcOut) { //get input rdd (needs to be longwritable/text for consistency with meta data, in case of //serialization issues create longwritableser/textser as serializable wrappers JavaPairRDD in = (JavaPairRDD) - sec.getRDDHandleForMatrixObject(sec.getMatrixObject(input1), FileFormat.CSV); - + sec.getRDDHandleForMatrixObject(sec.getMatrixObject(input1), FileFormat.CSV); + //reblock csv to binary block return RDDConverterUtils.csvToBinaryBlock(sec.getSparkContext(), - in, mcOut, _hasHeader, _delim, _fill, _fillValue, _naStrings); + in, mcOut, _hasHeader, _delim, _fill, _fillValue, _naStrings); } @SuppressWarnings("unchecked") - protected JavaPairRDD processFrameCSVReblockInstruction(SparkExecutionContext sec, DataCharacteristics mcOut, ValueType[] schema) { + protected JavaPairRDD processFrameCSVReblockInstruction(SparkExecutionContext sec, DataCharacteristics mcOut, ValueType[] schema) { //get input rdd (needs to be longwritable/text for consistency with meta data, in case of //serialization issues create longwritableser/textser as serializable wrappers - JavaPairRDD in = (JavaPairRDD) - sec.getRDDHandleForFrameObject(sec.getFrameObject(input1), FileFormat.CSV); - + JavaPairRDD in = (JavaPairRDD) + sec.getRDDHandleForFrameObject(sec.getFrameObject(input1), FileFormat.CSV); + //reblock csv to binary block return FrameRDDConverterUtils.csvToBinaryBlock(sec.getSparkContext(), - in, mcOut, schema, _hasHeader, _delim, _fill, _fillValue, _naStrings); + in, mcOut, schema, _hasHeader, _delim, _fill, _fillValue, _naStrings); } } diff --git a/src/main/java/org/apache/sysds/runtime/instructions/spark/FrameAppendRSPInstruction.java b/src/main/java/org/apache/sysds/runtime/instructions/spark/FrameAppendRSPInstruction.java index 8774c63ed7c..704f68ce8ea 100644 --- a/src/main/java/org/apache/sysds/runtime/instructions/spark/FrameAppendRSPInstruction.java +++ b/src/main/java/org/apache/sysds/runtime/instructions/spark/FrameAppendRSPInstruction.java @@ -28,6 +28,7 @@ import org.apache.sysds.runtime.controlprogram.context.SparkExecutionContext; import org.apache.sysds.runtime.frame.data.FrameBlock; import org.apache.sysds.runtime.instructions.cp.CPOperand; +import org.apache.sysds.runtime.instructions.spark.functions.SetColumnNamesFunction; import org.apache.sysds.runtime.instructions.spark.utils.FrameRDDAggregateUtils; import org.apache.sysds.runtime.matrix.operators.Operator; import scala.Tuple2; @@ -40,71 +41,115 @@ public class FrameAppendRSPInstruction extends AppendRSPInstruction { protected FrameAppendRSPInstruction(Operator op, CPOperand in1, CPOperand in2, CPOperand out, boolean cbind, - String opcode, String istr) { + String opcode, String istr) { super(op, in1, in2, out, cbind, opcode, istr); } @Override public void processInstruction(ExecutionContext ec) { - SparkExecutionContext sec = (SparkExecutionContext)ec; - JavaPairRDD in1 = sec.getFrameBinaryBlockRDDHandleForVariable( input1.getName() ); - JavaPairRDD in2 = sec.getFrameBinaryBlockRDDHandleForVariable( input2.getName() ); - JavaPairRDD out; + SparkExecutionContext sec = (SparkExecutionContext) ec; + JavaPairRDD in1 = sec.getFrameBinaryBlockRDDHandleForVariable(input1.getName()); + JavaPairRDD in2 = sec.getFrameBinaryBlockRDDHandleForVariable(input2.getName()); + JavaPairRDD out; long leftRows = sec.getDataCharacteristics(input1.getName()).getRows(); - out = appendFrameRSP(in1, in2, leftRows, _cbind); + String[] leftColumnNames = sec.getFrameObject(input1.getName()).getColumnNames(); + String[] rightColumnNames = sec.getFrameObject(input2.getName()).getColumnNames(); + String[] outputColumnNames = createOutputColumnNames(leftColumnNames, rightColumnNames, _cbind); //put output RDD handle into symbol table updateBinaryAppendOutputDataCharacteristics(sec, _cbind); + + sec.getFrameObject(output.getName()).setColumnNames(outputColumnNames); + + if (_cbind) { + //update schema and column names of output with merged input schemas + sec.getFrameObject(output.getName()).setSchema( + sec.getFrameObject(input1.getName()).mergeSchemas( + sec.getFrameObject(input2.getName()))); + } else { + sec.getFrameObject(output.getName()).setSchema(sec.getFrameObject(input1.getName()).getSchema()); + } + + out = appendFrameRSP(in1, in2, leftRows, _cbind); + + out = out.mapValues(new SetColumnNamesFunction(outputColumnNames)); + sec.setRDDHandleForVariable(output.getName(), out); sec.addLineageRDD(output.getName(), input1.getName()); sec.addLineageRDD(output.getName(), input2.getName()); - - if(_cbind) - //update schema of output with merged input schemas - sec.getFrameObject(output.getName()).setSchema( - sec.getFrameObject(input1.getName()).mergeSchemas( - sec.getFrameObject(input2.getName()))); - else - sec.getFrameObject(output.getName()).setSchema(sec.getFrameObject(input1.getName()).getSchema()); + } + + private static String[] createOutputColumnNames( + String[] leftNames, + String[] rightNames, + boolean cbind) { + if (leftNames == null) + return null; + + if (!cbind) + return leftNames.clone(); + + if (rightNames == null) + return null; + + String[] result = + new String[leftNames.length + rightNames.length]; + + System.arraycopy( + leftNames, + 0, + result, + 0, + leftNames.length); + + System.arraycopy( + rightNames, + 0, + result, + leftNames.length, + rightNames.length); + + return result; } public static JavaPairRDD appendFrameRSP(JavaPairRDD in1, JavaPairRDD in2, long leftRows, boolean cbind) { - if(cbind) { + if (cbind) { //TODO preserve info if already aligned, and only align if necessary //get in1 keys long[] row_indices = in1.keys().collect().stream().mapToLong(Long::longValue).toArray(); Arrays.sort(row_indices); //Align the blocks of in2 on the blocks of in1 - JavaPairRDD in2Aligned = in2.flatMapToPair(new ReduceSideAppendAlignToLHSFunction(row_indices, leftRows)); + JavaPairRDD in2Aligned = in2.flatMapToPair(new ReduceSideAppendAlignToLHSFunction(row_indices, leftRows)); in2Aligned = FrameRDDAggregateUtils.mergeByKey(in2Aligned); return in1.join(in2Aligned).mapValues(new ReduceSideColumnsFunction(cbind)); - } else { //rbind - JavaPairRDD right = in2.mapToPair( new ReduceSideAppendRowsFunction(leftRows)); + } else { //rbind + JavaPairRDD right = in2.mapToPair(new ReduceSideAppendRowsFunction(leftRows)); return in1.union(right); } } - private static class ReduceSideColumnsFunction implements Function, FrameBlock> - { + private static class ReduceSideColumnsFunction implements Function, FrameBlock> { private static final long serialVersionUID = -97824903649667646L; private boolean _cbind = true; - + public ReduceSideColumnsFunction(boolean cbind) { _cbind = cbind; } - + @Override - public FrameBlock call(Tuple2 arg0) { - FrameBlock left = arg0._1(); - FrameBlock right = arg0._2(); - return left.append(right, _cbind); + public FrameBlock call(Tuple2 input) { + FrameBlock left = input._1(); + FrameBlock right = input._2(); + + FrameBlock result = left.append(right, _cbind); + + return result; } } - private static class ReduceSideAppendAlignToLHSFunction implements PairFlatMapFunction, Long, FrameBlock> - { + private static class ReduceSideAppendAlignToLHSFunction implements PairFlatMapFunction, Long, FrameBlock> { private static final long serialVersionUID = 5850400295183766409L; private final long[] _indices; @@ -116,8 +161,7 @@ public ReduceSideAppendAlignToLHSFunction(long[] indices, long max_rows) { } @Override - public Iterator> call(Tuple2 arg0) - { + public Iterator> call(Tuple2 arg0) { List> aligned_blocks = new ArrayList<>(); long indexRHS = arg0._1(); FrameBlock fb = arg0._2(); @@ -127,15 +171,15 @@ public Iterator> call(Tuple2 arg0) int L = 0; int R = _indices.length - 1; int m; - while(L <= R){ + while (L <= R) { m = (L + R) / 2; - if(_indices[m] == indexRHS){ + if (_indices[m] == indexRHS) { R = m; break; } - if(_indices[m] < indexRHS) + if (_indices[m] < indexRHS) L = m + 1; - else + else R = m - 1; } // search terminates if we have found the exact indexRHS or binary search reached the leaf nodes where @@ -147,8 +191,8 @@ public Iterator> call(Tuple2 arg0) long indexLHS = _indices[R]; //assumes total num rows LHS == RHS - long nextIndexLHS = R < _indices.length - 1? _indices[R+1] : this.lastIndex; - int blkSizeLHS = (int) (nextIndexLHS - indexLHS); + long nextIndexLHS = R < _indices.length - 1 ? _indices[R + 1] : this.lastIndex; + int blkSizeLHS = (int) (nextIndexLHS - indexLHS); int offsetLHS = (int) (indexRHS - indexLHS); int offsetRHS = 0; int sizeOfSlice = blkSizeLHS - offsetLHS; @@ -157,70 +201,66 @@ public Iterator> call(Tuple2 arg0) resultBlock.ensureAllocatedColumns(blkSizeLHS); int sizeOfRHS = fb.getNumRows(); - while(sizeOfSlice < sizeOfRHS){ + while (sizeOfSlice < sizeOfRHS) { FrameBlock fb_sliced = fb.slice(offsetRHS, offsetRHS + sizeOfSlice - 1); - resultBlock = resultBlock.leftIndexingOperations(fb_sliced,offsetLHS, offsetLHS + sizeOfSlice - 1, 0, fb.getNumColumns()-1, new FrameBlock()); + resultBlock = resultBlock.leftIndexingOperations(fb_sliced, offsetLHS, offsetLHS + sizeOfSlice - 1, 0, fb.getNumColumns() - 1, new FrameBlock()); aligned_blocks.add(new Tuple2<>(indexLHS, resultBlock)); resultBlock = new FrameBlock(fb.getSchema()); - if(R >= _indices.length - 1) + if (R >= _indices.length - 1) throw new RuntimeException("Alignment Error while CBIND: LHS has fewer rows than RHS"); indexLHS = nextIndexLHS; offsetRHS += sizeOfSlice; offsetLHS = 0; sizeOfRHS -= sizeOfSlice; R++; - nextIndexLHS = R < _indices.length - 1? _indices[R+1] : this.lastIndex; - sizeOfSlice = (int) (nextIndexLHS - indexLHS); //sizeOfSlice = blkSizeLHS + nextIndexLHS = R < _indices.length - 1 ? _indices[R + 1] : this.lastIndex; + sizeOfSlice = (int) (nextIndexLHS - indexLHS); //sizeOfSlice = blkSizeLHS resultBlock.ensureAllocatedColumns(sizeOfSlice); } //RHS fits into aligned LHS block - if(offsetRHS != 0) + if (offsetRHS != 0) fb = fb.slice(offsetRHS, offsetRHS + sizeOfRHS - 1); - resultBlock = resultBlock.leftIndexingOperations(fb, offsetLHS, offsetLHS + fb.getNumRows() - 1, 0, fb.getNumColumns()-1, new FrameBlock()); + resultBlock = resultBlock.leftIndexingOperations(fb, offsetLHS, offsetLHS + fb.getNumRows() - 1, 0, fb.getNumColumns() - 1, new FrameBlock()); aligned_blocks.add(new Tuple2<>(indexLHS, resultBlock)); return aligned_blocks.iterator(); } } - private static class ReduceSideAppendRowsFunction implements PairFunction, Long, FrameBlock> - { + private static class ReduceSideAppendRowsFunction implements PairFunction, Long, FrameBlock> { private static final long serialVersionUID = 1723795153048336791L; private long _offset; - + public ReduceSideAppendRowsFunction(long offset) { _offset = offset; } - + @Override - public Tuple2 call(Tuple2 arg0) - throws Exception - { - return new Tuple2<>(arg0._1()+_offset, arg0._2()); + public Tuple2 call(Tuple2 arg0) + throws Exception { + return new Tuple2<>(arg0._1() + _offset, arg0._2()); } } @SuppressWarnings("unused") - private static class ReduceSideAppendAlignFunction implements PairFunction, Long, FrameBlock> - { + private static class ReduceSideAppendAlignFunction implements PairFunction, Long, FrameBlock> { private static final long serialVersionUID = 5850400295183766409L; private long _rows; - + public ReduceSideAppendAlignFunction(long rows) { _rows = rows; } - + @Override - public Tuple2 call(Tuple2 arg0) - throws Exception - { + public Tuple2 call(Tuple2 arg0) + throws Exception { FrameBlock resultBlock = new FrameBlock(arg0._2().getSchema()); - long index = (arg0._1()/OptimizerUtils.DEFAULT_FRAME_BLOCKSIZE)*OptimizerUtils.DEFAULT_FRAME_BLOCKSIZE+1; - int maxRows = (int) (_rows - index+1 >= OptimizerUtils.DEFAULT_FRAME_BLOCKSIZE?OptimizerUtils.DEFAULT_FRAME_BLOCKSIZE:_rows - index+1); + long index = (arg0._1() / OptimizerUtils.DEFAULT_FRAME_BLOCKSIZE) * OptimizerUtils.DEFAULT_FRAME_BLOCKSIZE + 1; + int maxRows = (int) (_rows - index + 1 >= OptimizerUtils.DEFAULT_FRAME_BLOCKSIZE ? OptimizerUtils.DEFAULT_FRAME_BLOCKSIZE : _rows - index + 1); resultBlock.ensureAllocatedColumns(maxRows); - resultBlock = resultBlock.leftIndexingOperations(arg0._2(), 0, maxRows-1, 0, arg0._2().getNumColumns()-1, new FrameBlock()); + resultBlock = resultBlock.leftIndexingOperations(arg0._2(), 0, maxRows - 1, 0, arg0._2().getNumColumns() - 1, new FrameBlock()); return new Tuple2<>(index, resultBlock); } } diff --git a/src/main/java/org/apache/sysds/runtime/instructions/spark/FrameIndexingSPInstruction.java b/src/main/java/org/apache/sysds/runtime/instructions/spark/FrameIndexingSPInstruction.java index 0dc768d5328..98c34ced282 100644 --- a/src/main/java/org/apache/sysds/runtime/instructions/spark/FrameIndexingSPInstruction.java +++ b/src/main/java/org/apache/sysds/runtime/instructions/spark/FrameIndexingSPInstruction.java @@ -101,6 +101,11 @@ public void processInstruction(ExecutionContext ec) { //update schema of output with subset of input schema sec.getFrameObject(output.getName()).setSchema( sec.getFrameObject(input1.getName()).getSchema((int)cl, (int)cu)); + + // update column names of output with subset of input column names + sec.getFrameObject(output.getName()).setColumnNames( + sec.getFrameObject(input1.getName()).getColumnNames((int)cl, (int)cu)); + } //left indexing else if ( opcode.equalsIgnoreCase(Opcodes.LEFT_INDEX.toString()) || opcode.equalsIgnoreCase(Opcodes.MAPLEFTINDEX.toString())) diff --git a/src/main/java/org/apache/sysds/runtime/instructions/spark/ReblockSPInstruction.java b/src/main/java/org/apache/sysds/runtime/instructions/spark/ReblockSPInstruction.java index 170190f6b87..e38a495d01e 100644 --- a/src/main/java/org/apache/sysds/runtime/instructions/spark/ReblockSPInstruction.java +++ b/src/main/java/org/apache/sysds/runtime/instructions/spark/ReblockSPInstruction.java @@ -59,7 +59,7 @@ public class ReblockSPInstruction extends UnarySPInstruction { private boolean outputEmptyBlocks; private ReblockSPInstruction(Operator op, CPOperand in, CPOperand out, int br, int bc, boolean emptyBlocks, - String opcode, String instr) { + String opcode, String instr) { super(SPType.Reblock, op, in, out, opcode, instr); blen = br; blen = bc; @@ -70,13 +70,13 @@ public static ReblockSPInstruction parseInstruction(String str) { String parts[] = InstructionUtils.getInstructionPartsWithValueType(str); String opcode = parts[0]; - if(!opcode.equals(Opcodes.RBLK.toString())) { + if (!opcode.equals(Opcodes.RBLK.toString())) { throw new DMLRuntimeException("Incorrect opcode for ReblockSPInstruction:" + opcode); } CPOperand in = new CPOperand(parts[1]); CPOperand out = new CPOperand(parts[2]); - int blen=Integer.parseInt(parts[3]); + int blen = Integer.parseInt(parts[3]); boolean outputEmptyBlocks = Boolean.parseBoolean(parts[4]); Operator op = null; // no operator for ReblockSPInstruction @@ -85,7 +85,7 @@ public static ReblockSPInstruction parseInstruction(String str) { @Override public void processInstruction(ExecutionContext ec) { - SparkExecutionContext sec = (SparkExecutionContext)ec; + SparkExecutionContext sec = (SparkExecutionContext) ec; //set the output characteristics CacheableData obj = sec.getCacheableData(input1.getName()); @@ -95,23 +95,29 @@ public void processInstruction(ExecutionContext ec) { //get the source format from the meta data MetaDataFormat iimd = (MetaDataFormat) obj.getMetaData(); - if(iimd == null) + if (iimd == null) throw new DMLRuntimeException("Error: Metadata not found"); + if (input1.getDataType() == DataType.FRAME) { + FrameObject inputFrame = sec.getFrameObject(input1.getName()); + FrameObject outputFrame = sec.getFrameObject(output.getName()); + outputFrame.setColumnNames(inputFrame.getColumnNames()); + } + //check for in-memory reblock (w/ lazy spark context, potential for latency reduction) - if( Recompiler.checkCPReblock(sec, input1.getName()) ) { - if( input1.getDataType().isMatrix() || input1.getDataType().isFrame() ) { + if (Recompiler.checkCPReblock(sec, input1.getName())) { + if (input1.getDataType().isMatrix() || input1.getDataType().isFrame()) { Recompiler.executeInMemoryReblock(sec, input1.getName(), output.getName(), - iimd.getFileFormat()==FileFormat.BINARY ? getLineageItem(ec).getValue() : null); + iimd.getFileFormat() == FileFormat.BINARY ? getLineageItem(ec).getValue() : null); } Statistics.decrementNoOfExecutedSPInst(); return; } //execute matrix/frame reblock - if( input1.getDataType() == DataType.MATRIX ) + if (input1.getDataType() == DataType.MATRIX) processMatrixReblockInstruction(sec, iimd.getFileFormat()); - else if(input1.getDataType() == DataType.FRAME) + else if (input1.getDataType() == DataType.FRAME) processFrameReblockInstruction(sec, iimd.getFileFormat()); } @@ -121,24 +127,23 @@ protected void processMatrixReblockInstruction(SparkExecutionContext sec, FileFo DataCharacteristics mc = sec.getDataCharacteristics(input1.getName()); DataCharacteristics mcOut = sec.getDataCharacteristics(output.getName()); - if(fmt == FileFormat.TEXT || fmt == FileFormat.MM ) { + if (fmt == FileFormat.TEXT || fmt == FileFormat.MM) { //get matrix market file properties if necessary FileFormatPropertiesMM mmProps = (fmt == FileFormat.MM) ? - IOUtilFunctions.readAndParseMatrixMarketHeader(mo.getFileName()) : null; + IOUtilFunctions.readAndParseMatrixMarketHeader(mo.getFileName()) : null; //get the input textcell rdd JavaPairRDD lines = (JavaPairRDD) - sec.getRDDHandleForMatrixObject(mo, fmt); + sec.getRDDHandleForMatrixObject(mo, fmt); //convert textcell to binary block JavaPairRDD out = RDDConverterUtils.textCellToBinaryBlock( - sec.getSparkContext(), lines, mcOut, outputEmptyBlocks, mmProps); + sec.getSparkContext(), lines, mcOut, outputEmptyBlocks, mmProps); //put output RDD handle into symbol table sec.setRDDHandleForVariable(output.getName(), out); sec.addLineageRDD(output.getName(), input1.getName()); - } - else if(fmt == FileFormat.CSV) { + } else if (fmt == FileFormat.CSV) { // HACK ALERT: Until we introduces the rewrite to insert csvrblock for non-persistent read // throw new DMLRuntimeException("CSVInputInfo is not supported for ReblockSPInstruction"); CSVReblockSPInstruction csvInstruction = null; @@ -147,9 +152,8 @@ else if(fmt == FileFormat.CSV) { boolean fill = false; double fillValue = 0; Set naStrings = null; - if(mo.getFileFormatProperties() instanceof FileFormatPropertiesCSV - && mo.getFileFormatProperties() != null ) - { + if (mo.getFileFormatProperties() instanceof FileFormatPropertiesCSV + && mo.getFileFormatProperties() != null) { FileFormatPropertiesCSV props = (FileFormatPropertiesCSV) mo.getFileFormatProperties(); hasHeader = props.hasHeader(); delim = props.getDelim(); @@ -161,8 +165,7 @@ else if(fmt == FileFormat.CSV) { csvInstruction = new CSVReblockSPInstruction(null, input1, output, mcOut.getBlocksize(), mcOut.getBlocksize(), hasHeader, delim, fill, fillValue, Opcodes.CSVRBLK.toString(), instString, naStrings); csvInstruction.processInstruction(sec); return; - } - else if(fmt == FileFormat.BINARY && mc.getBlocksize() <= 0) { + } else if (fmt == FileFormat.BINARY && mc.getBlocksize() <= 0) { //BINARY BLOCK <- BINARY CELL (e.g., after grouped aggregate) JavaPairRDD binaryCells = (JavaPairRDD) sec.getRDDHandleForMatrixObject(mo, FileFormat.BINARY); JavaPairRDD out = RDDConverterUtils.binaryCellToBinaryBlock(sec.getSparkContext(), binaryCells, mcOut, outputEmptyBlocks); @@ -170,63 +173,57 @@ else if(fmt == FileFormat.BINARY && mc.getBlocksize() <= 0) { //put output RDD handle into symbol table sec.setRDDHandleForVariable(output.getName(), out); sec.addLineageRDD(output.getName(), input1.getName()); - } - else if(fmt == FileFormat.BINARY) { + } else if (fmt == FileFormat.BINARY) { //BINARY BLOCK <- BINARY BLOCK (different sizes) JavaPairRDD in1 = sec.getBinaryMatrixBlockRDDHandleForVariable(input1.getName()); JavaPairRDD out = RDDConverterUtils.binaryBlockToBinaryBlock(in1, mc, mcOut); - + //put output RDD handle into symbol table sec.setRDDHandleForVariable(output.getName(), out); sec.addLineageRDD(output.getName(), input1.getName()); - } - else if(fmt == FileFormat.LIBSVM) { + } else if (fmt == FileFormat.LIBSVM) { String delim = IOUtilFunctions.LIBSVM_DELIM; String indexDelim = IOUtilFunctions.LIBSVM_INDEX_DELIM; - if(mo.getFileFormatProperties() instanceof FileFormatPropertiesLIBSVM && mo - .getFileFormatProperties() != null) { + if (mo.getFileFormatProperties() instanceof FileFormatPropertiesLIBSVM && mo + .getFileFormatProperties() != null) { FileFormatPropertiesLIBSVM props = (FileFormatPropertiesLIBSVM) mo.getFileFormatProperties(); delim = props.getDelim(); indexDelim = props.getIndexDelim(); } LIBSVMReblockSPInstruction libsvmInstruction = new LIBSVMReblockSPInstruction(null, input1, output, - mcOut.getBlocksize(), mcOut.getBlocksize(), "libsvmblk", delim, indexDelim, instString); + mcOut.getBlocksize(), mcOut.getBlocksize(), "libsvmblk", delim, indexDelim, instString); libsvmInstruction.processInstruction(sec); - } - else if(fmt == FileFormat.COMPRESSED){ + } else if (fmt == FileFormat.COMPRESSED) { JavaPairRDD in1 = (JavaPairRDD) sec - .getRDDHandleForMatrixObject(mo, FileFormat.COMPRESSED); + .getRDDHandleForMatrixObject(mo, FileFormat.COMPRESSED); JavaPairRDD out = RDDConverterUtils.binaryBlockToBinaryBlock(in1, mc, mcOut); sec.setRDDHandleForVariable(output.getName(), out); sec.addLineageRDD(output.getName(), input1.getName()); - } - else { + } else { throw new DMLRuntimeException("The given format is not implemented " - + "for ReblockSPInstruction:" + fmt.toString()); + + "for ReblockSPInstruction:" + fmt.toString()); } } @SuppressWarnings("unchecked") - protected void processFrameReblockInstruction(SparkExecutionContext sec, FileFormat fmt) - { + protected void processFrameReblockInstruction(SparkExecutionContext sec, FileFormat fmt) { FrameObject fo = sec.getFrameObject(input1.getName()); DataCharacteristics mcOut = sec.getDataCharacteristics(output.getName()); - if(fmt == FileFormat.TEXT) { + if (fmt == FileFormat.TEXT) { //get the input textcell rdd JavaPairRDD lines = (JavaPairRDD) - sec.getRDDHandleForFrameObject(fo, fmt); + sec.getRDDHandleForFrameObject(fo, fmt); //convert textcell to binary block JavaPairRDD out = - FrameRDDConverterUtils.textCellToBinaryBlock(sec.getSparkContext(), lines, mcOut, fo.getSchema()); + FrameRDDConverterUtils.textCellToBinaryBlock(sec.getSparkContext(), lines, mcOut, fo.getSchema()); //put output RDD handle into symbol table sec.setRDDHandleForVariable(output.getName(), out); sec.addLineageRDD(output.getName(), input1.getName()); - } - else if(fmt == FileFormat.CSV) { + } else if (fmt == FileFormat.CSV) { // HACK ALERT: Until we introduces the rewrite to insert csvrblock for non-persistent read // throw new DMLRuntimeException("CSVInputInfo is not supported for ReblockSPInstruction"); CSVReblockSPInstruction csvInstruction = null; @@ -235,9 +232,8 @@ else if(fmt == FileFormat.CSV) { boolean fill = false; double fillValue = 0; Set naStrings = null; - if(fo.getFileFormatProperties() instanceof FileFormatPropertiesCSV - && fo.getFileFormatProperties() != null ) - { + if (fo.getFileFormatProperties() instanceof FileFormatPropertiesCSV + && fo.getFileFormatProperties() != null) { FileFormatPropertiesCSV props = (FileFormatPropertiesCSV) fo.getFileFormatProperties(); hasHeader = props.hasHeader(); delim = props.getDelim(); @@ -248,34 +244,31 @@ else if(fmt == FileFormat.CSV) { csvInstruction = new CSVReblockSPInstruction(null, input1, output, mcOut.getBlocksize(), mcOut.getBlocksize(), hasHeader, delim, fill, fillValue, Opcodes.CSVRBLK.toString(), instString, naStrings); csvInstruction.processInstruction(sec); - } - else if(fmt == FileFormat.LIBSVM) { + } else if (fmt == FileFormat.LIBSVM) { String delim = IOUtilFunctions.LIBSVM_DELIM; String indexDelim = IOUtilFunctions.LIBSVM_INDEX_DELIM; - if(fo.getFileFormatProperties() instanceof FileFormatPropertiesLIBSVM && fo - .getFileFormatProperties() != null) { + if (fo.getFileFormatProperties() instanceof FileFormatPropertiesLIBSVM && fo + .getFileFormatProperties() != null) { FileFormatPropertiesLIBSVM props = (FileFormatPropertiesLIBSVM) fo.getFileFormatProperties(); delim = props.getDelim(); indexDelim = props.getIndexDelim(); } LIBSVMReblockSPInstruction libsvmInstruction = new LIBSVMReblockSPInstruction(null, input1, output, - mcOut.getBlocksize(), mcOut.getBlocksize(), "libsvmblk", delim, indexDelim, instString); + mcOut.getBlocksize(), mcOut.getBlocksize(), "libsvmblk", delim, indexDelim, instString); libsvmInstruction.processInstruction(sec); - } - - else { + } else { throw new DMLRuntimeException("The given format is not implemented " - + "for ReblockSPInstruction: " + fmt.toString()); + + "for ReblockSPInstruction: " + fmt.toString()); } } - + @Override public Pair getLineageItem(ExecutionContext ec) { //construct reblock lineage without existing createvar lineage - if( ec.getLineage() == null ) { + if (ec.getLineage() == null) { return Pair.of(output.getName(), new LineageItem( - ProgramConverter.serializeDataObject(input1.getName(), ec.getCacheableData(input1)), "cache_rblk")); + ProgramConverter.serializeDataObject(input1.getName(), ec.getCacheableData(input1)), "cache_rblk")); } //default reblock w/ active lineage tracing return super.getLineageItem(ec); diff --git a/src/main/java/org/apache/sysds/runtime/instructions/spark/functions/SetColumnNamesFunction.java b/src/main/java/org/apache/sysds/runtime/instructions/spark/functions/SetColumnNamesFunction.java new file mode 100644 index 00000000000..6ea874da572 --- /dev/null +++ b/src/main/java/org/apache/sysds/runtime/instructions/spark/functions/SetColumnNamesFunction.java @@ -0,0 +1,46 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + + +package org.apache.sysds.runtime.instructions.spark.functions; + +import org.apache.spark.api.java.function.Function; +import org.apache.sysds.runtime.frame.data.FrameBlock; + +public class SetColumnNamesFunction implements Function { + private static final long serialVersionUID = 1L; + + private final String[] _columnNames; + + public SetColumnNamesFunction(String[] columnNames) { + _columnNames = columnNames != null + ? columnNames.clone() + : null; + } + + @Override + public FrameBlock call(FrameBlock block) { + block.setColumnNames( + _columnNames != null + ? _columnNames.clone() + : null); + + return block; + } +} diff --git a/src/main/java/org/apache/sysds/runtime/instructions/spark/utils/FrameRDDConverterUtils.java b/src/main/java/org/apache/sysds/runtime/instructions/spark/utils/FrameRDDConverterUtils.java index 9371d43094c..3a35e43a98c 100644 --- a/src/main/java/org/apache/sysds/runtime/instructions/spark/utils/FrameRDDConverterUtils.java +++ b/src/main/java/org/apache/sysds/runtime/instructions/spark/utils/FrameRDDConverterUtils.java @@ -663,9 +663,9 @@ public Iterator call(Tuple2 arg0) //handle header information and frame meta data if( ix==1 ) { if( _props.hasHeader() ) { - for(int j = 1; j <= blk.getNumColumns(); j++) { - sb.append(blk.getColumnNames()[j] - + ((j reader = + informat.getRecordReader( + splits[0], + job, + Reporter.NULL + ); + + LongWritable key = new LongWritable(); + Text value = new Text(); + + try { + if(!reader.next(key, value)) + throw new IOException( + "CSV frame does not contain a header: " + fname + ); + + return value.toString().split( + Pattern.quote(_props.getDelim()), + -1 + ); + } + finally { + IOUtilFunctions.closeSilently(reader); + } + } + catch(IOException ex) { + throw new DMLRuntimeException( + "Failed to read CSV header from: " + fname, + ex + ); + } + } + @Override public final FrameBlock readFrameFromHDFS(String fname, ValueType[] schema, String[] names, long rlen, long clen) throws IOException, DMLRuntimeException { @@ -86,6 +138,8 @@ public final FrameBlock readFrameFromHDFS(String fname, ValueType[] schema, Stri return ret; } + + @Override public FrameBlock readFrameFromInputStream(InputStream is, ValueType[] schema, String[] names, long rlen, long clen) throws IOException, DMLRuntimeException diff --git a/src/main/java/org/apache/sysds/runtime/util/ProgramConverter.java b/src/main/java/org/apache/sysds/runtime/util/ProgramConverter.java index b0ef37280f9..350c13070cb 100644 --- a/src/main/java/org/apache/sysds/runtime/util/ProgramConverter.java +++ b/src/main/java/org/apache/sysds/runtime/util/ProgramConverter.java @@ -99,42 +99,36 @@ import org.apache.sysds.runtime.meta.MetaDataFormat; import org.apache.sysds.utils.stats.InfrastructureAnalyzer; -import java.util.ArrayList; -import java.util.HashMap; -import java.util.HashSet; -import java.util.List; -import java.util.Set; -import java.util.StringTokenizer; +import java.util.*; import java.util.stream.Collectors; /** - * Program converter functionalities for - * (1) creating deep copies of program blocks, instructions, function program blocks, and - * (2) serializing and parsing of programs, program blocks, functions program blocks. - * + * Program converter functionalities for + * (1) creating deep copies of program blocks, instructions, function program blocks, and + * (2) serializing and parsing of programs, program blocks, functions program blocks. + * */ //TODO: rewrite class to instance-based invocation (grown gradually and now inappropriate design) -public class ProgramConverter -{ +public class ProgramConverter { protected static final Log LOG = LogFactory.getLog(ProgramConverter.class.getName()); //use escaped unicodes for separators in order to prevent string conflict - public static final String NEWLINE = "\n"; //System.lineSeparator(); - public static final String COMPONENTS_DELIM = "\u236e"; //semicolon w/ bar; ";"; - public static final String ELEMENT_DELIM = "\u236a"; //comma w/ bar; ","; - public static final String ELEMENT_DELIM2 = ","; - public static final String DATA_FIELD_DELIM = "\u007c"; //"|"; - public static final String KEY_VALUE_DELIM = "\u003d"; //"="; - public static final String LEVELIN = "\u23a8"; //variant of left curly bracket; "\u007b"; //"{"; - public static final String LEVELOUT = "\u23ac"; //variant of right curly bracket; "\u007d"; //"}"; - public static final String EMPTY = "null"; - public static final String DASH = "-"; - public static final String REF = "ref"; + public static final String NEWLINE = "\n"; //System.lineSeparator(); + public static final String COMPONENTS_DELIM = "\u236e"; //semicolon w/ bar; ";"; + public static final String ELEMENT_DELIM = "\u236a"; //comma w/ bar; ","; + public static final String ELEMENT_DELIM2 = ","; + public static final String DATA_FIELD_DELIM = "\u007c"; //"|"; + public static final String KEY_VALUE_DELIM = "\u003d"; //"="; + public static final String LEVELIN = "\u23a8"; //variant of left curly bracket; "\u007b"; //"{"; + public static final String LEVELOUT = "\u23ac"; //variant of right curly bracket; "\u007d"; //"}"; + public static final String EMPTY = "null"; + public static final String DASH = "-"; + public static final String REF = "ref"; public static final String LIST_ELEMENT_DELIM = "\t"; public static final String CDATA_BEGIN = ""; - + public static final String PROG_BEGIN = " PROG" + LEVELIN; public static final String PROG_END = LEVELOUT; public static final String VARS_BEGIN = "VARS: "; @@ -157,125 +151,118 @@ public class ProgramConverter public static final String CONF_STATS = "stats"; // Used for parfor - public static final String PARFORBODY_BEGIN = CDATA_BEGIN + "PARFORBODY" + LEVELIN; - public static final String PARFORBODY_END = LEVELOUT + CDATA_END; + public static final String PARFORBODY_BEGIN = CDATA_BEGIN + "PARFORBODY" + LEVELIN; + public static final String PARFORBODY_END = LEVELOUT + CDATA_END; // Used for paramserv builtin function public static final String PSBODY_BEGIN = CDATA_BEGIN + "PSBODY" + LEVELIN; public static final String PSBODY_END = LEVELOUT + CDATA_END; - + //exception msgs - public static final String NOT_SUPPORTED_SPARK_INSTRUCTION = "Not supported: Instructions of type other than CP instructions"; - public static final String NOT_SUPPORTED_SPARK_PARFOR = "Not supported: Nested ParFOR REMOTE_SPARK due to possible deadlocks." + - "(LOCAL can be used for innner ParFOR)"; - public static final String NOT_SUPPORTED_PB = "Not supported: type of program block"; - + public static final String NOT_SUPPORTED_SPARK_INSTRUCTION = "Not supported: Instructions of type other than CP instructions"; + public static final String NOT_SUPPORTED_SPARK_PARFOR = "Not supported: Nested ParFOR REMOTE_SPARK due to possible deadlocks." + + "(LOCAL can be used for innner ParFOR)"; + public static final String NOT_SUPPORTED_PB = "Not supported: type of program block"; + //////////////////////////////// // CREATION of DEEP COPIES //////////////////////////////// - + /** * Creates a deep copy of the given execution context. * For rt_platform=Hadoop, execution context has a symbol table. - * + * * @param ec execution context * @return execution context * @throws CloneNotSupportedException if CloneNotSupportedException occurs */ - public static ExecutionContext createDeepCopyExecutionContext(ExecutionContext ec) - throws CloneNotSupportedException - { + public static ExecutionContext createDeepCopyExecutionContext(ExecutionContext ec) + throws CloneNotSupportedException { ExecutionContext cpec = ExecutionContextFactory.createContext(false, ec.getProgram()); cpec.setVariables((LocalVariableMap) ec.getVariables().clone()); - if( ec.getLineage() != null ) + if (ec.getLineage() != null) cpec.setLineage(new Lineage(ec.getLineage())); - + //handle result variables with in-place update flag //(each worker requires its own copy of the empty matrix object) - for( String var : cpec.getVariables().keySet() ) { + for (String var : cpec.getVariables().keySet()) { Data dat = cpec.getVariables().get(var); - if( dat instanceof MatrixObject && ((MatrixObject)dat).getUpdateType().isInPlace() ) { - MatrixObject mo = (MatrixObject)dat; - MatrixObject moNew = new MatrixObject(mo); - if( mo.getNnz() != 0 ){ + if (dat instanceof MatrixObject && ((MatrixObject) dat).getUpdateType().isInPlace()) { + MatrixObject mo = (MatrixObject) dat; + MatrixObject moNew = new MatrixObject(mo); + if (mo.getNnz() != 0) { // If output matrix is not empty (NNZ != 0), then local copy is created so that // update in place operation can be applied. MatrixBlock mbVar = mo.acquireRead(); - moNew.acquireModify (new MatrixBlock(mbVar)); + moNew.acquireModify(new MatrixBlock(mbVar)); mo.release(); } else { //create empty matrix block w/ dense representation (preferred for update in-place) //Creating a dense matrix block is valid because empty block not allocated and transfer // to sparse representation happens in left indexing in place operation. - moNew.acquireModify(new MatrixBlock((int)mo.getNumRows(), (int)mo.getNumColumns(), false)); + moNew.acquireModify(new MatrixBlock((int) mo.getNumRows(), (int) mo.getNumColumns(), false)); } moNew.release(); cpec.setVariable(var, moNew); } } - + return cpec; } - + /** * This recursively creates a deep copy of program blocks and transparently replaces filenames according to the * specified parallel worker in order to avoid conflicts between parworkers. This happens recursively in order - * to support arbitrary control-flow constructs within a parfor. - * - * @param childBlocks child program blocks - * @param pid ? - * @param IDPrefix ? - * @param fnStack ? - * @param fnCreated ? - * @param plain if true, full deep copy without id replacement + * to support arbitrary control-flow constructs within a parfor. + * + * @param childBlocks child program blocks + * @param pid ? + * @param IDPrefix ? + * @param fnStack ? + * @param fnCreated ? + * @param plain if true, full deep copy without id replacement * @param forceDeepCopy if true, force deep copy * @return list of program blocks */ - public static ArrayList rcreateDeepCopyProgramBlocks(ArrayList childBlocks, long pid, int IDPrefix, Set fnStack, Set fnCreated, boolean plain, boolean forceDeepCopy) - { + public static ArrayList rcreateDeepCopyProgramBlocks(ArrayList childBlocks, long pid, int IDPrefix, Set fnStack, Set fnCreated, boolean plain, boolean forceDeepCopy) { ArrayList tmp = new ArrayList<>(); - - for( ProgramBlock pb : childBlocks ) - { + + for (ProgramBlock pb : childBlocks) { Program prog = pb.getProgram(); ProgramBlock tmpPB = null; - - if( pb instanceof WhileProgramBlock ) { + + if (pb instanceof WhileProgramBlock) { tmpPB = createDeepCopyWhileProgramBlock((WhileProgramBlock) pb, pid, IDPrefix, prog, fnStack, fnCreated, plain, forceDeepCopy); - } - else if( pb instanceof ForProgramBlock && !(pb instanceof ParForProgramBlock) ) { - tmpPB = createDeepCopyForProgramBlock((ForProgramBlock) pb, pid, IDPrefix, prog, fnStack, fnCreated, plain, forceDeepCopy ); - } - else if( pb instanceof ParForProgramBlock ) { + } else if (pb instanceof ForProgramBlock && !(pb instanceof ParForProgramBlock)) { + tmpPB = createDeepCopyForProgramBlock((ForProgramBlock) pb, pid, IDPrefix, prog, fnStack, fnCreated, plain, forceDeepCopy); + } else if (pb instanceof ParForProgramBlock) { ParForProgramBlock pfpb = (ParForProgramBlock) pb; - if( ParForProgramBlock.ALLOW_NESTED_PARALLELISM ) + if (ParForProgramBlock.ALLOW_NESTED_PARALLELISM) tmpPB = createDeepCopyParForProgramBlock(pfpb, pid, IDPrefix, prog, fnStack, fnCreated, plain, forceDeepCopy); - else + else tmpPB = createDeepCopyForProgramBlock((ForProgramBlock) pb, pid, IDPrefix, prog, fnStack, fnCreated, plain, forceDeepCopy); - } - else if( pb instanceof IfProgramBlock ) { + } else if (pb instanceof IfProgramBlock) { tmpPB = createDeepCopyIfProgramBlock((IfProgramBlock) pb, pid, IDPrefix, prog, fnStack, fnCreated, plain, forceDeepCopy); - } - else if( pb instanceof BasicProgramBlock ) { //last-level program block + } else if (pb instanceof BasicProgramBlock) { //last-level program block BasicProgramBlock bpb = (BasicProgramBlock) pb; tmpPB = new BasicProgramBlock(prog); // general case use for most PBs - + //for recompile in the master node JVM - tmpPB.setStatementBlock(createStatementBlockCopy(bpb.getStatementBlock(), pid, plain, forceDeepCopy)); + tmpPB.setStatementBlock(createStatementBlockCopy(bpb.getStatementBlock(), pid, plain, forceDeepCopy)); tmpPB.setThreadID(pid); - + //copy instructions - ((BasicProgramBlock)tmpPB).setInstructions( - createDeepCopyInstructionSet(bpb.getInstructions(), - pid, IDPrefix, prog, fnStack, fnCreated, plain, true) ); + ((BasicProgramBlock) tmpPB).setInstructions( + createDeepCopyInstructionSet(bpb.getInstructions(), + pid, IDPrefix, prog, fnStack, fnCreated, plain, true)); } - + //copy symbol table //tmpPB.setVariables( pb.getVariables() ); //implicit cloning - + tmp.add(tmpPB); } - + return tmp; } @@ -283,8 +270,8 @@ public static WhileProgramBlock createDeepCopyWhileProgramBlock(WhileProgramBloc ArrayList predinst = createDeepCopyInstructionSet(wpb.getPredicate(), pid, IDPrefix, prog, fnStack, fnCreated, plain, true); WhileProgramBlock tmpPB = new WhileProgramBlock(prog, predinst); StatementBlock sb = ConfigurationManager.getCompilerConfigFlag(ConfigType.ALLOW_PARALLEL_DYN_RECOMPILATION) ? - createWhileStatementBlockCopy((WhileStatementBlock) wpb.getStatementBlock(), forceDeepCopy) : wpb.getStatementBlock(); - tmpPB.setStatementBlock( sb ); + createWhileStatementBlockCopy((WhileStatementBlock) wpb.getStatementBlock(), forceDeepCopy) : wpb.getStatementBlock(); + tmpPB.setStatementBlock(sb); tmpPB.setThreadID(pid); tmpPB.setChildBlocks(rcreateDeepCopyProgramBlocks(wpb.getChildBlocks(), pid, IDPrefix, fnStack, fnCreated, plain, forceDeepCopy)); tmpPB.setExitInstruction(wpb.getExitInstruction()); @@ -295,8 +282,8 @@ public static IfProgramBlock createDeepCopyIfProgramBlock(IfProgramBlock ipb, lo ArrayList predinst = createDeepCopyInstructionSet(ipb.getPredicate(), pid, IDPrefix, prog, fnStack, fnCreated, plain, true); IfProgramBlock tmpPB = new IfProgramBlock(prog, predinst); StatementBlock sb = ConfigurationManager.getCompilerConfigFlag(ConfigType.ALLOW_PARALLEL_DYN_RECOMPILATION) ? - createIfStatementBlockCopy((IfStatementBlock)ipb.getStatementBlock(), forceDeepCopy ) : ipb.getStatementBlock(); - tmpPB.setStatementBlock( sb ); + createIfStatementBlockCopy((IfStatementBlock) ipb.getStatementBlock(), forceDeepCopy) : ipb.getStatementBlock(); + tmpPB.setStatementBlock(sb); tmpPB.setThreadID(pid); tmpPB.setChildBlocksIfBody(rcreateDeepCopyProgramBlocks(ipb.getChildBlocksIfBody(), pid, IDPrefix, fnStack, fnCreated, plain, forceDeepCopy)); tmpPB.setChildBlocksElseBody(rcreateDeepCopyProgramBlocks(ipb.getChildBlocksElseBody(), pid, IDPrefix, fnStack, fnCreated, plain, forceDeepCopy)); @@ -305,103 +292,101 @@ public static IfProgramBlock createDeepCopyIfProgramBlock(IfProgramBlock ipb, lo } public static ForProgramBlock createDeepCopyForProgramBlock(ForProgramBlock fpb, long pid, int IDPrefix, Program prog, Set fnStack, Set fnCreated, boolean plain, boolean forceDeepCopy) { - ForProgramBlock tmpPB = new ForProgramBlock(prog,fpb.getIterVar()); + ForProgramBlock tmpPB = new ForProgramBlock(prog, fpb.getIterVar()); StatementBlock sb = ConfigurationManager.getCompilerConfigFlag(ConfigType.ALLOW_PARALLEL_DYN_RECOMPILATION) ? - createForStatementBlockCopy((ForStatementBlock)fpb.getStatementBlock(), forceDeepCopy) : fpb.getStatementBlock(); + createForStatementBlockCopy((ForStatementBlock) fpb.getStatementBlock(), forceDeepCopy) : fpb.getStatementBlock(); tmpPB.setStatementBlock(sb); tmpPB.setThreadID(pid); - tmpPB.setFromInstructions( createDeepCopyInstructionSet(fpb.getFromInstructions(), pid, IDPrefix, prog, fnStack, fnCreated, plain, true) ); - tmpPB.setToInstructions( createDeepCopyInstructionSet(fpb.getToInstructions(), pid, IDPrefix, prog, fnStack, fnCreated, plain, true) ); - tmpPB.setIncrementInstructions( createDeepCopyInstructionSet(fpb.getIncrementInstructions(), pid, IDPrefix, prog, fnStack, fnCreated, plain, true) ); - tmpPB.setChildBlocks( rcreateDeepCopyProgramBlocks(fpb.getChildBlocks(), pid, IDPrefix, fnStack, fnCreated, plain, forceDeepCopy) ); + tmpPB.setFromInstructions(createDeepCopyInstructionSet(fpb.getFromInstructions(), pid, IDPrefix, prog, fnStack, fnCreated, plain, true)); + tmpPB.setToInstructions(createDeepCopyInstructionSet(fpb.getToInstructions(), pid, IDPrefix, prog, fnStack, fnCreated, plain, true)); + tmpPB.setIncrementInstructions(createDeepCopyInstructionSet(fpb.getIncrementInstructions(), pid, IDPrefix, prog, fnStack, fnCreated, plain, true)); + tmpPB.setChildBlocks(rcreateDeepCopyProgramBlocks(fpb.getChildBlocks(), pid, IDPrefix, fnStack, fnCreated, plain, forceDeepCopy)); tmpPB.setExitInstruction(fpb.getExitInstruction()); return tmpPB; } - public static ForProgramBlock createShallowCopyForProgramBlock(ForProgramBlock fpb, Program prog ) { - ForProgramBlock tmpPB = new ForProgramBlock(prog,fpb.getIterVar()); - tmpPB.setFromInstructions( fpb.getFromInstructions() ); - tmpPB.setToInstructions( fpb.getToInstructions() ); - tmpPB.setIncrementInstructions( fpb.getIncrementInstructions() ); - tmpPB.setChildBlocks( fpb.getChildBlocks() ); + public static ForProgramBlock createShallowCopyForProgramBlock(ForProgramBlock fpb, Program prog) { + ForProgramBlock tmpPB = new ForProgramBlock(prog, fpb.getIterVar()); + tmpPB.setFromInstructions(fpb.getFromInstructions()); + tmpPB.setToInstructions(fpb.getToInstructions()); + tmpPB.setIncrementInstructions(fpb.getIncrementInstructions()); + tmpPB.setChildBlocks(fpb.getChildBlocks()); tmpPB.setExitInstruction(fpb.getExitInstruction()); return tmpPB; } public static ParForProgramBlock createDeepCopyParForProgramBlock(ParForProgramBlock pfpb, long pid, int IDPrefix, Program prog, Set fnStack, Set fnCreated, boolean plain, boolean forceDeepCopy) { ParForProgramBlock tmpPB = null; - - if( IDPrefix == -1 ) //still on master node - tmpPB = new ParForProgramBlock(prog,pfpb.getIterVar(), pfpb.getParForParams(), pfpb.getResultVariables()); + + if (IDPrefix == -1) //still on master node + tmpPB = new ParForProgramBlock(prog, pfpb.getIterVar(), pfpb.getParForParams(), pfpb.getResultVariables()); else //child of remote ParWorker at any level tmpPB = new ParForProgramBlock(IDPrefix, prog, pfpb.getIterVar(), pfpb.getParForParams(), pfpb.getResultVariables()); - + StatementBlock sb = ConfigurationManager.getCompilerConfigFlag(ConfigType.ALLOW_PARALLEL_DYN_RECOMPILATION) ? - createForStatementBlockCopy((ForStatementBlock)pfpb.getStatementBlock(), forceDeepCopy) : pfpb.getStatementBlock(); - tmpPB.setStatementBlock( sb ); + createForStatementBlockCopy((ForStatementBlock) pfpb.getStatementBlock(), forceDeepCopy) : pfpb.getStatementBlock(); + tmpPB.setStatementBlock(sb); tmpPB.setThreadID(pid); - + tmpPB.disableOptimization(); //already done in top-level parfor - - tmpPB.setFromInstructions( createDeepCopyInstructionSet(pfpb.getFromInstructions(), pid, IDPrefix, prog, fnStack, fnCreated, plain, true) ); - tmpPB.setToInstructions( createDeepCopyInstructionSet(pfpb.getToInstructions(), pid, IDPrefix, prog, fnStack, fnCreated, plain, true) ); - tmpPB.setIncrementInstructions( createDeepCopyInstructionSet(pfpb.getIncrementInstructions(), pid, IDPrefix, prog, fnStack, fnCreated, plain, true) ); - + + tmpPB.setFromInstructions(createDeepCopyInstructionSet(pfpb.getFromInstructions(), pid, IDPrefix, prog, fnStack, fnCreated, plain, true)); + tmpPB.setToInstructions(createDeepCopyInstructionSet(pfpb.getToInstructions(), pid, IDPrefix, prog, fnStack, fnCreated, plain, true)); + tmpPB.setIncrementInstructions(createDeepCopyInstructionSet(pfpb.getIncrementInstructions(), pid, IDPrefix, prog, fnStack, fnCreated, plain, true)); + //NOTE: Normally, no recursive copy because (1) copied on each execution in this PB anyway //and (2) leave placeholders as they are. However, if plain, an explicit deep copy is requested. - if( plain || forceDeepCopy ) - tmpPB.setChildBlocks( rcreateDeepCopyProgramBlocks(pfpb.getChildBlocks(), pid, IDPrefix, fnStack, fnCreated, plain, forceDeepCopy) ); + if (plain || forceDeepCopy) + tmpPB.setChildBlocks(rcreateDeepCopyProgramBlocks(pfpb.getChildBlocks(), pid, IDPrefix, fnStack, fnCreated, plain, forceDeepCopy)); else - tmpPB.setChildBlocks( pfpb.getChildBlocks() ); + tmpPB.setChildBlocks(pfpb.getChildBlocks()); tmpPB.setExitInstruction(pfpb.getExitInstruction()); - + return tmpPB; } - + /** * This creates a deep copy of a function program block. The central reference to singletons of function program blocks * poses the need for explicit copies in order to prevent conflicting writes of temporary variables (see ExternalFunctionProgramBlock. - * + * * @param namespace function namespace - * @param oldName ? - * @param pid ? - * @param IDPrefix ? - * @param prog runtime program - * @param fnStack ? + * @param oldName ? + * @param pid ? + * @param IDPrefix ? + * @param prog runtime program + * @param fnStack ? * @param fnCreated ? - * @param plain ? + * @param plain ? */ - public static void createDeepCopyFunctionProgramBlock(String namespace, String oldName, long pid, int IDPrefix, Program prog, Set fnStack, Set fnCreated, boolean plain) - { + public static void createDeepCopyFunctionProgramBlock(String namespace, String oldName, long pid, int IDPrefix, Program prog, Set fnStack, Set fnCreated, boolean plain) { //fpb guaranteed to be non-null (checked inside getFunctionProgramBlock) FunctionProgramBlock fpb1 = prog.getFunctionProgramBlock(namespace, oldName, true); FunctionProgramBlock fpb2 = prog.containsFunctionProgramBlock(namespace, oldName, false) ? - prog.getFunctionProgramBlock(namespace, oldName, false) : null; - String fnameNew = (plain)? oldName :(oldName+Lop.CP_CHILD_THREAD+pid); - String fnameNewKey = DMLProgram.constructFunctionKey(namespace,fnameNew); + prog.getFunctionProgramBlock(namespace, oldName, false) : null; + String fnameNew = (plain) ? oldName : (oldName + Lop.CP_CHILD_THREAD + pid); + String fnameNewKey = DMLProgram.constructFunctionKey(namespace, fnameNew); - if( prog.getFunctionProgramBlocks().containsKey(fnameNewKey) ) + if (prog.getFunctionProgramBlocks().containsKey(fnameNewKey)) return; //prevent redundant deep copy if already existent - + //create deep copy FunctionProgramBlock copy1 = null; - if( !fnStack.contains(fnameNewKey) ) { + if (!fnStack.contains(fnameNewKey)) { fnStack.add(fnameNewKey); copy1 = createDeepCopyFunctionProgramBlock(fpb1, fnStack, fnCreated, pid, IDPrefix, plain); fnStack.remove(fnameNewKey); - } - else //stop deep copy for recursive function calls + } else //stop deep copy for recursive function calls copy1 = fpb1; - + //copy.setVariables( (LocalVariableMap) fpb.getVariables() ); //implicit cloning //note: instructions not used by function program block - + //put if not existing (recursive processing might have added it) - if( !prog.getFunctionProgramBlocks().containsKey(fnameNewKey) ) { + if (!prog.getFunctionProgramBlocks().containsKey(fnameNewKey)) { prog.addFunctionProgramBlock(namespace, fnameNew, copy1, true); - if( fpb2 != null ) { + if (fpb2 != null) { FunctionProgramBlock copy2 = createDeepCopyFunctionProgramBlock( - fpb2, fnStack, fnCreated, pid, IDPrefix, plain); + fpb2, fnStack, fnCreated, pid, IDPrefix, plain); prog.addFunctionProgramBlock(namespace, fnameNew, copy2, false); } fnCreated.add(DMLProgram.constructFunctionKey(namespace, fnameNew)); @@ -411,105 +396,100 @@ public static void createDeepCopyFunctionProgramBlock(String namespace, String o public static FunctionProgramBlock createDeepCopyFunctionProgramBlock(FunctionProgramBlock fpb, Set fnStack, Set fnCreated) { return createDeepCopyFunctionProgramBlock(fpb, fnStack, fnCreated, 0, -1, true); } - + public static FunctionProgramBlock createDeepCopyFunctionProgramBlock(FunctionProgramBlock fpb, Set fnStack, Set fnCreated, long pid) { //recursive deep copy with creation of thread-specific function calls return createDeepCopyFunctionProgramBlock(fpb, fnStack, fnCreated, pid, -1, false); } - - public static FunctionProgramBlock createDeepCopyFunctionProgramBlock(FunctionProgramBlock fpb, Set fnStack, Set fnCreated, long pid, int IDPrefix, boolean plain) - { - if( fpb == null ) + + public static FunctionProgramBlock createDeepCopyFunctionProgramBlock(FunctionProgramBlock fpb, Set fnStack, Set fnCreated, long pid, int IDPrefix, boolean plain) { + if (fpb == null) throw new DMLRuntimeException("Unable to create a deep copy of a non-existing FunctionProgramBlock."); - + //create deep copy FunctionProgramBlock copy = null; ArrayList tmp1 = new ArrayList<>(); ArrayList tmp2 = new ArrayList<>(); - if( fpb.getInputParams()!= null ) + if (fpb.getInputParams() != null) tmp1.addAll(fpb.getInputParams()); - if( fpb.getOutputParams()!= null ) + if (fpb.getOutputParams() != null) tmp2.addAll(fpb.getOutputParams()); - + copy = new FunctionProgramBlock(fpb.getProgram(), tmp1, tmp2); - copy.setChildBlocks( rcreateDeepCopyProgramBlocks(fpb.getChildBlocks(), pid, IDPrefix, fnStack, fnCreated, plain, fpb.isRecompileOnce()) ); - copy.setStatementBlock( fpb.getStatementBlock() ); + copy.setChildBlocks(rcreateDeepCopyProgramBlocks(fpb.getChildBlocks(), pid, IDPrefix, fnStack, fnCreated, plain, fpb.isRecompileOnce())); + copy.setStatementBlock(fpb.getStatementBlock()); copy.setRecompileOnce(fpb.isRecompileOnce()); copy.setThreadID(pid); - + return copy; } - + /** * Creates a deep copy of an array of instructions and replaces the placeholders of parworker * IDs with the concrete IDs of this parfor instance. This is a helper method uses for generating * deep copies of program blocks. - * - * @param instSet list of instructions - * @param pid ? - * @param IDPrefix ? - * @param prog runtime program - * @param fnStack ? - * @param fnCreated ? - * @param plain ? + * + * @param instSet list of instructions + * @param pid ? + * @param IDPrefix ? + * @param prog runtime program + * @param fnStack ? + * @param fnCreated ? + * @param plain ? * @param cpFunctions ? * @return list of instructions */ public static ArrayList createDeepCopyInstructionSet(ArrayList instSet, long pid, int IDPrefix, Program prog, Set fnStack, Set fnCreated, boolean plain, boolean cpFunctions) { ArrayList tmp = new ArrayList<>(); - for( Instruction inst : instSet ) { - if( inst instanceof FunctionCallCPInstruction && cpFunctions ) { + for (Instruction inst : instSet) { + if (inst instanceof FunctionCallCPInstruction && cpFunctions) { FunctionCallCPInstruction finst = (FunctionCallCPInstruction) inst; - createDeepCopyFunctionProgramBlock( finst.getNamespace(), - finst.getFunctionName(), pid, IDPrefix, prog, fnStack, fnCreated, plain ); + createDeepCopyFunctionProgramBlock(finst.getNamespace(), + finst.getFunctionName(), pid, IDPrefix, prog, fnStack, fnCreated, plain); } - tmp.add( cloneInstruction( inst, pid, plain, cpFunctions ) ); + tmp.add(cloneInstruction(inst, pid, plain, cpFunctions)); } return tmp; } public static ArrayList createShallowCopyInstructionSet(ArrayList insts, long pid) { ArrayList ret = new ArrayList<>(); - for( Instruction inst : insts ) { + for (Instruction inst : insts) { //save replacement of thread id references in instructions - ret.add(saveReplaceThreadID( inst, Lop.CP_ROOT_THREAD_ID, Lop.CP_CHILD_THREAD+pid)); + ret.add(saveReplaceThreadID(inst, Lop.CP_ROOT_THREAD_ID, Lop.CP_CHILD_THREAD + pid)); } return ret; } - - public static Instruction cloneInstruction( Instruction oInst, long pid, boolean plain, boolean cpFunctions ) - { + + public static Instruction cloneInstruction(Instruction oInst, long pid, boolean plain, boolean cpFunctions) { Instruction inst = null; String tmpString = oInst.toString(); - - try - { - if( oInst instanceof CPInstruction || oInst instanceof SPInstruction || oInst instanceof FEDInstruction - || oInst instanceof GPUInstruction || oInst instanceof OOCInstruction ) { - if( oInst instanceof FunctionCallCPInstruction && cpFunctions ) { + + try { + if (oInst instanceof CPInstruction || oInst instanceof SPInstruction || oInst instanceof FEDInstruction + || oInst instanceof GPUInstruction || oInst instanceof OOCInstruction) { + if (oInst instanceof FunctionCallCPInstruction && cpFunctions) { FunctionCallCPInstruction tmp = (FunctionCallCPInstruction) oInst; - if( !plain ) { + if (!plain) { //safe replacement because target variables might include the function name //note: this is no update-in-place in order to keep the original function name as basis - tmpString = tmp.updateInstStringFunctionName(tmp.getFunctionName(), tmp.getFunctionName() + Lop.CP_CHILD_THREAD+pid); + tmpString = tmp.updateInstStringFunctionName(tmp.getFunctionName(), tmp.getFunctionName() + Lop.CP_CHILD_THREAD + pid); } //otherwise: preserve function name } inst = InstructionParser.parseSingleInstruction(tmpString); - } - else - throw new DMLRuntimeException("Failed to clone instruction: "+oInst); - } - catch(Exception ex) { + } else + throw new DMLRuntimeException("Failed to clone instruction: " + oInst); + } catch (Exception ex) { throw new DMLRuntimeException(ex); } - + //save replacement of thread id references in instructions - inst = saveReplaceThreadID( inst, Lop.CP_ROOT_THREAD_ID, Lop.CP_CHILD_THREAD+pid); - + inst = saveReplaceThreadID(inst, Lop.CP_ROOT_THREAD_ID, Lop.CP_CHILD_THREAD + pid); + return inst; } - + public static FunctionStatementBlock createDeepCopyFunctionStatementBlock(FunctionStatementBlock fsb, Set fnStack, Set fnCreated) { FunctionStatement fstmt = (FunctionStatement) fsb.getStatement(0); FunctionStatementBlock retSb = new FunctionStatementBlock(); @@ -521,232 +501,210 @@ public static FunctionStatementBlock createDeepCopyFunctionStatementBlock(Functi retSb.addStatement(retStmt); retSb.setDMLProg(fsb.getDMLProg()); retSb.setParseInfo(fsb); - retSb.setLiveIn( fsb.liveIn() ); - retSb.setLiveOut( fsb.liveOut() ); - for( StatementBlock sb : fstmt.getBody() ) + retSb.setLiveIn(fsb.liveIn()); + retSb.setLiveOut(fsb.liveOut()); + for (StatementBlock sb : fstmt.getBody()) retStmt.getBody().add(rCreateDeepCopyStatementBlock(sb)); return retSb; } - + public static StatementBlock rCreateDeepCopyStatementBlock(StatementBlock sb) { StatementBlock ret = null; - if( sb instanceof IfStatementBlock ) { + if (sb instanceof IfStatementBlock) { IfStatementBlock orig = (IfStatementBlock) sb; IfStatementBlock isb = createIfStatementBlockCopy(orig, true); IfStatement origstmt = (IfStatement) orig.getStatement(0); IfStatement istmt = new IfStatement(); //only shallow istmt.setConditionalPredicate(origstmt.getConditionalPredicate()); isb.setStatements(CollectionUtils.asArrayList(istmt)); - for( StatementBlock c : origstmt.getIfBody() ) + for (StatementBlock c : origstmt.getIfBody()) istmt.addStatementBlockIfBody(rCreateDeepCopyStatementBlock(c)); - for( StatementBlock c : origstmt.getElseBody() ) + for (StatementBlock c : origstmt.getElseBody()) istmt.addStatementBlockElseBody(rCreateDeepCopyStatementBlock(c)); ret = isb; - } - else if( sb instanceof WhileStatementBlock ) { + } else if (sb instanceof WhileStatementBlock) { WhileStatementBlock orig = (WhileStatementBlock) sb; WhileStatementBlock wsb = createWhileStatementBlockCopy(orig, true); WhileStatement origstmt = (WhileStatement) orig.getStatement(0); WhileStatement wstmt = new WhileStatement(); //only shallow wstmt.setPredicate(origstmt.getConditionalPredicate()); wsb.setStatements(CollectionUtils.asArrayList(wstmt)); - for( StatementBlock c : origstmt.getBody() ) + for (StatementBlock c : origstmt.getBody()) wstmt.addStatementBlock(rCreateDeepCopyStatementBlock(c)); ret = wsb; - } - else if( sb instanceof ForStatementBlock ) { //incl parfor + } else if (sb instanceof ForStatementBlock) { //incl parfor ForStatementBlock orig = (ForStatementBlock) sb; ForStatementBlock fsb = createForStatementBlockCopy(orig, true); ForStatement origstmt = (ForStatement) orig.getStatement(0); ForStatement fstmt = (origstmt instanceof ParForStatement) ? - new ParForStatement() : new ForStatement(); //only shallow + new ParForStatement() : new ForStatement(); //only shallow fstmt.setPredicate(origstmt.getIterablePredicate()); fsb.setStatements(CollectionUtils.asArrayList(fstmt)); - for( StatementBlock c : origstmt.getBody() ) + for (StatementBlock c : origstmt.getBody()) fstmt.addStatementBlock(rCreateDeepCopyStatementBlock(c)); ret = fsb; - } - else { + } else { StatementBlock bsb = createStatementBlockCopy(sb, -1, true, true); - for( Hop root : bsb.getHops() ) - if( root instanceof FunctionOp ) - ((FunctionOp)root).setCallOptimized(false); + for (Hop root : bsb.getHops()) + if (root instanceof FunctionOp) + ((FunctionOp) root).setCallOptimized(false); ret = bsb; } return ret; } - public static StatementBlock createStatementBlockCopy( StatementBlock sb, long pid, boolean plain, boolean forceDeepCopy ) - { + public static StatementBlock createStatementBlockCopy(StatementBlock sb, long pid, boolean plain, boolean forceDeepCopy) { StatementBlock ret = null; - - try - { - if( sb != null //forced deep copy for function recompilation - && (Recompiler.requiresRecompilation( sb.getHops() ) || forceDeepCopy) ) - { + + try { + if (sb != null //forced deep copy for function recompilation + && (Recompiler.requiresRecompilation(sb.getHops()) || forceDeepCopy)) { //create new statement (shallow copy livein/liveout for recompile, line numbers for explain) ret = new StatementBlock(); ret.setDMLProg(sb.getDMLProg()); ret.setParseInfo(sb); - ret.setLiveIn( sb.liveIn() ); - ret.setLiveOut( sb.liveOut() ); - ret.setUpdatedVariables( sb.variablesUpdated() ); - ret.setReadVariables( sb.variablesRead() ); - + ret.setLiveIn(sb.liveIn()); + ret.setLiveOut(sb.liveOut()); + ret.setUpdatedVariables(sb.variablesUpdated()); + ret.setReadVariables(sb.variablesRead()); + //deep copy hops dag for concurrent recompile ArrayList hops = sb.getHops(); - synchronized(hops) { // guard concurrent recompile - hops = Recompiler.deepCopyHopsDag( hops ); + synchronized (hops) { // guard concurrent recompile + hops = Recompiler.deepCopyHopsDag(hops); } - if( !plain ) - Recompiler.updateFunctionNames( hops, pid ); - ret.setHops( hops ); + if (!plain) + Recompiler.updateFunctionNames(hops, pid); + ret.setHops(hops); ret.updateRecompilationFlag(); ret.setNondeterministic(sb.isNondeterministic()); - } - else { + } else { ret = sb; } + } catch (Exception ex) { + throw new DMLRuntimeException(ex); } - catch( Exception ex ) { - throw new DMLRuntimeException( ex ); - } - + return ret; } - public static IfStatementBlock createIfStatementBlockCopy( IfStatementBlock sb, boolean forceDeepCopy ) - { + public static IfStatementBlock createIfStatementBlockCopy(IfStatementBlock sb, boolean forceDeepCopy) { IfStatementBlock ret = null; - - try - { - if( sb != null //forced deep copy for function recompile - && (Recompiler.requiresRecompilation( sb.getPredicateHops() ) || forceDeepCopy) ) - { + + try { + if (sb != null //forced deep copy for function recompile + && (Recompiler.requiresRecompilation(sb.getPredicateHops()) || forceDeepCopy)) { //create new statement (shallow copy livein/liveout for recompile, line numbers for explain) ret = new IfStatementBlock(); ret.setDMLProg(sb.getDMLProg()); ret.setParseInfo(sb); - ret.setLiveIn( sb.liveIn() ); - ret.setLiveOut( sb.liveOut() ); - ret.setUpdatedVariables( sb.variablesUpdated() ); - ret.setReadVariables( sb.variablesRead() ); - + ret.setLiveIn(sb.liveIn()); + ret.setLiveOut(sb.liveOut()); + ret.setUpdatedVariables(sb.variablesUpdated()); + ret.setReadVariables(sb.variablesRead()); + //shallow copy child statements - ret.setStatements( sb.getStatements() ); - + ret.setStatements(sb.getStatements()); + //deep copy predicate hops dag for concurrent recompile - Hop hops = Recompiler.deepCopyHopsDag( sb.getPredicateHops() ); - ret.setPredicateHops( hops ); + Hop hops = Recompiler.deepCopyHopsDag(sb.getPredicateHops()); + ret.setPredicateHops(hops); ret.updatePredicateRecompilationFlag(); ret.setNondeterministic(sb.isNondeterministic()); - } - else { + } else { ret = sb; } + } catch (Exception ex) { + throw new DMLRuntimeException(ex); } - catch( Exception ex ) { - throw new DMLRuntimeException( ex ); - } - + return ret; } - public static WhileStatementBlock createWhileStatementBlockCopy( WhileStatementBlock sb, boolean forceDeepCopy ) - { + public static WhileStatementBlock createWhileStatementBlockCopy(WhileStatementBlock sb, boolean forceDeepCopy) { WhileStatementBlock ret = null; - - try - { - if( sb != null //forced deep copy for function recompile - && (Recompiler.requiresRecompilation( sb.getPredicateHops() ) || forceDeepCopy) ) - { + + try { + if (sb != null //forced deep copy for function recompile + && (Recompiler.requiresRecompilation(sb.getPredicateHops()) || forceDeepCopy)) { //create new statement (shallow copy livein/liveout for recompile, line numbers for explain) ret = new WhileStatementBlock(); ret.setDMLProg(sb.getDMLProg()); ret.setParseInfo(sb); - ret.setLiveIn( sb.liveIn() ); - ret.setLiveOut( sb.liveOut() ); - ret.setUpdatedVariables( sb.variablesUpdated() ); - ret.setReadVariables( sb.variablesRead() ); - ret.setUpdateInPlaceVars( sb.getUpdateInPlaceVars() ); - ret.setRecompileOnce( sb.isRecompileOnce() ); - + ret.setLiveIn(sb.liveIn()); + ret.setLiveOut(sb.liveOut()); + ret.setUpdatedVariables(sb.variablesUpdated()); + ret.setReadVariables(sb.variablesRead()); + ret.setUpdateInPlaceVars(sb.getUpdateInPlaceVars()); + ret.setRecompileOnce(sb.isRecompileOnce()); + //shallow copy child statements - ret.setStatements( sb.getStatements() ); - + ret.setStatements(sb.getStatements()); + //deep copy predicate hops dag for concurrent recompile - Hop hops = Recompiler.deepCopyHopsDag( sb.getPredicateHops() ); - ret.setPredicateHops( hops ); + Hop hops = Recompiler.deepCopyHopsDag(sb.getPredicateHops()); + ret.setPredicateHops(hops); ret.updatePredicateRecompilationFlag(); ret.setNondeterministic(sb.isNondeterministic()); - } - else { + } else { ret = sb; } + } catch (Exception ex) { + throw new DMLRuntimeException(ex); } - catch( Exception ex ) { - throw new DMLRuntimeException( ex ); - } - + return ret; } - public static ForStatementBlock createForStatementBlockCopy( ForStatementBlock sb, boolean forceDeepCopy ) - { + public static ForStatementBlock createForStatementBlockCopy(ForStatementBlock sb, boolean forceDeepCopy) { ForStatementBlock ret = null; - - try - { - if( sb != null && (forceDeepCopy - || Recompiler.requiresRecompilation(sb.getFromHops()) - || Recompiler.requiresRecompilation(sb.getToHops()) - || Recompiler.requiresRecompilation(sb.getIncrementHops())) ) - { + + try { + if (sb != null && (forceDeepCopy + || Recompiler.requiresRecompilation(sb.getFromHops()) + || Recompiler.requiresRecompilation(sb.getToHops()) + || Recompiler.requiresRecompilation(sb.getIncrementHops()))) { ret = (sb instanceof ParForStatementBlock) ? new ParForStatementBlock() : new ForStatementBlock(); - + //create new statement (shallow copy livein/liveout for recompile, line numbers for explain) ret.setDMLProg(sb.getDMLProg()); ret.setParseInfo(sb); - ret.setLiveIn( sb.liveIn() ); - ret.setLiveOut( sb.liveOut() ); - ret.setUpdatedVariables( sb.variablesUpdated() ); - ret.setReadVariables( sb.variablesRead() ); - ret.setUpdateInPlaceVars( sb.getUpdateInPlaceVars() ); - ret.setRecompileOnce( sb.isRecompileOnce() ); - + ret.setLiveIn(sb.liveIn()); + ret.setLiveOut(sb.liveOut()); + ret.setUpdatedVariables(sb.variablesUpdated()); + ret.setReadVariables(sb.variablesRead()); + ret.setUpdateInPlaceVars(sb.getUpdateInPlaceVars()); + ret.setRecompileOnce(sb.isRecompileOnce()); + //shallow copy child statements - ret.setStatements( sb.getStatements() ); - + ret.setStatements(sb.getStatements()); + //deep copy predicate hops dag for concurrent recompile //or on create full statement block copies - ret.setFromHops( Recompiler.deepCopyHopsDag(sb.getFromHops())); + ret.setFromHops(Recompiler.deepCopyHopsDag(sb.getFromHops())); ret.setToHops(Recompiler.deepCopyHopsDag(sb.getToHops())); - if( sb.getIncrementHops() != null ) + if (sb.getIncrementHops() != null) ret.setIncrementHops(Recompiler.deepCopyHopsDag(sb.getIncrementHops())); - + ret.updatePredicateRecompilationFlags(); ret.setNondeterministic(sb.isNondeterministic()); - if( sb instanceof ParForStatementBlock ) - ((ParForStatementBlock)ret).setResultVariables(((ParForStatementBlock)sb).getResultVariables()); - } - else { + if (sb instanceof ParForStatementBlock) + ((ParForStatementBlock) ret).setResultVariables(((ParForStatementBlock) sb).getResultVariables()); + } else { ret = sb; } + } catch (Exception ex) { + throw new DMLRuntimeException(ex); } - catch( Exception ex ) { - throw new DMLRuntimeException( ex ); - } - + return ret; } - - + + //////////////////////////////// // SERIALIZATION - //////////////////////////////// + + /// ///////////////////////////// public static String serializeSparkPSBody(SparkPSBody body, HashMap clsMap) { @@ -774,7 +732,7 @@ public static String serializeSparkPSBody(SparkPSBody body, HashMap(ec.getProgram().getFunctionProgramBlocks().keySet()), clsMap)); + new HashSet<>(ec.getProgram().getFunctionProgramBlocks().keySet()), clsMap)); builder.append(PROG_END); builder.append(NEWLINE); builder.append(COMPONENTS_DELIM); @@ -801,76 +759,75 @@ public static String serializeSparkPSBody(SparkPSBody body, HashMap()); - } - - public static String serializeParForBody( ParForBody body, HashMap clsMap ) - { + } + + public static String serializeParForBody(ParForBody body, HashMap clsMap) { ArrayList pbs = body.getChildBlocks(); ArrayList rVnames = body.getResultVariables(); ExecutionContext ec = body.getEc(); - - if( pbs.isEmpty() ) + + if (pbs.isEmpty()) return PARFORBODY_BEGIN + PARFORBODY_END; - - Program prog = pbs.get( 0 ).getProgram(); - + + Program prog = pbs.get(0).getProgram(); + StringBuilder sb = new StringBuilder(); - sb.append( PARFORBODY_BEGIN ); - sb.append( NEWLINE ); - + sb.append(PARFORBODY_BEGIN); + sb.append(NEWLINE); + //handle DMLScript UUID (propagate original uuid for writing to scratch space) - sb.append( DMLScript.getUUID() ); - sb.append( COMPONENTS_DELIM ); - sb.append( NEWLINE ); - + sb.append(DMLScript.getUUID()); + sb.append(COMPONENTS_DELIM); + sb.append(NEWLINE); + //handle DML config - sb.append( ConfigurationManager.getDMLConfig().serializeDMLConfig() ); - sb.append( COMPONENTS_DELIM ); - sb.append( NEWLINE ); - + sb.append(ConfigurationManager.getDMLConfig().serializeDMLConfig()); + sb.append(COMPONENTS_DELIM); + sb.append(NEWLINE); + //handle additional configurations - sb.append( CONF_STATS + "=" + DMLScript.STATISTICS ); - sb.append( COMPONENTS_DELIM ); - sb.append( NEWLINE ); - + sb.append(CONF_STATS + "=" + DMLScript.STATISTICS); + sb.append(COMPONENTS_DELIM); + sb.append(NEWLINE); + //handle program sb.append(PROG_BEGIN); - sb.append( NEWLINE ); - sb.append( serializeProgram(prog, pbs, clsMap) ); + sb.append(NEWLINE); + sb.append(serializeProgram(prog, pbs, clsMap)); sb.append(PROG_END); - sb.append( NEWLINE ); - sb.append( COMPONENTS_DELIM ); - sb.append( NEWLINE ); - + sb.append(NEWLINE); + sb.append(COMPONENTS_DELIM); + sb.append(NEWLINE); + //handle result variable names - sb.append( serializeResultVariables(rVnames) ); - sb.append( COMPONENTS_DELIM ); - + sb.append(serializeResultVariables(rVnames)); + sb.append(COMPONENTS_DELIM); + //handle execution context //note: this includes also the symbol table (serialize only the top-level variable map, // (symbol tables for nested/child blocks are created at parse time, on the remote side) sb.append(EC_BEGIN); - sb.append( serializeExecutionContext(ec) ); + sb.append(serializeExecutionContext(ec)); sb.append(EC_END); - sb.append( NEWLINE ); - sb.append( COMPONENTS_DELIM ); - sb.append( NEWLINE ); - + sb.append(NEWLINE); + sb.append(COMPONENTS_DELIM); + sb.append(NEWLINE); + //handle program blocks sb.append(PBS_BEGIN); - sb.append( NEWLINE ); - sb.append( rSerializeProgramBlocks(pbs, clsMap) ); + sb.append(NEWLINE); + sb.append(rSerializeProgramBlocks(pbs, clsMap)); sb.append(PBS_END); - sb.append( NEWLINE ); - - sb.append( PARFORBODY_END ); - + sb.append(NEWLINE); + + sb.append(PARFORBODY_END); + return sb.toString(); } - public static String serializeProgram( Program prog, ArrayList pbs, HashMap clsMap) { + public static String serializeProgram(Program prog, ArrayList pbs, HashMap clsMap) { //note program contains variables, programblocks and function program blocks //but in order to avoid redundancy, we only serialize function program blocks HashSet cand = new HashSet<>(); @@ -878,59 +835,52 @@ public static String serializeProgram( Program prog, ArrayList pbs return rSerializeFunctionProgramBlocks(prog, cand, clsMap); } - private static void rFindSerializationCandidates( ArrayList pbs, HashSet cand) - { - for( ProgramBlock pb : pbs ) - { - if( pb instanceof WhileProgramBlock ) { + private static void rFindSerializationCandidates(ArrayList pbs, HashSet cand) { + for (ProgramBlock pb : pbs) { + if (pb instanceof WhileProgramBlock) { WhileProgramBlock wpb = (WhileProgramBlock) pb; rFindSerializationCandidates(wpb.getChildBlocks(), cand); - } - else if ( pb instanceof ForProgramBlock || pb instanceof ParForProgramBlock ) { - ForProgramBlock fpb = (ForProgramBlock) pb; + } else if (pb instanceof ForProgramBlock || pb instanceof ParForProgramBlock) { + ForProgramBlock fpb = (ForProgramBlock) pb; rFindSerializationCandidates(fpb.getChildBlocks(), cand); - } - else if ( pb instanceof IfProgramBlock ) { + } else if (pb instanceof IfProgramBlock) { IfProgramBlock ipb = (IfProgramBlock) pb; rFindSerializationCandidates(ipb.getChildBlocksIfBody(), cand); - if( ipb.getChildBlocksElseBody() != null ) + if (ipb.getChildBlocksElseBody() != null) rFindSerializationCandidates(ipb.getChildBlocksElseBody(), cand); - } - else if( pb instanceof BasicProgramBlock ) { + } else if (pb instanceof BasicProgramBlock) { BasicProgramBlock bpb = (BasicProgramBlock) pb; - for( Instruction inst : bpb.getInstructions() ) { - if( inst instanceof FunctionCallCPInstruction ) { + for (Instruction inst : bpb.getInstructions()) { + if (inst instanceof FunctionCallCPInstruction) { FunctionCallCPInstruction fci = (FunctionCallCPInstruction) inst; String fkey = DMLProgram.constructFunctionKey(fci.getNamespace(), fci.getFunctionName()); - if( !cand.contains(fkey) ) { //memoization for multiple calls, recursion - cand.add( fkey ); //add to candidates + if (!cand.contains(fkey)) { //memoization for multiple calls, recursion + cand.add(fkey); //add to candidates //investigate chains of function calls FunctionProgramBlock fpb = pb.getProgram().getFunctionProgramBlock(fci.getNamespace(), fci.getFunctionName()); rFindSerializationCandidates(fpb.getChildBlocks(), cand); } - } - else if(inst instanceof EvalNaryCPInstruction) { + } else if (inst instanceof EvalNaryCPInstruction) { //add all potential targets, included loaded builtin functions because other //functions might call them directly (not through eval and thus cannot be loaded) //(even if fname is a known literal, the target function might call other functions) pb.getProgram().getFunctionProgramBlocks().keySet().stream() - .forEach(s -> cand.add(s)); + .forEach(s -> cand.add(s)); } } } } } - private static String serializeVariables (LocalVariableMap vars) { + private static String serializeVariables(LocalVariableMap vars) { StringBuilder sb = new StringBuilder(); sb.append(VARS_BEGIN); - sb.append( vars.serialize() ); + sb.append(vars.serialize()); sb.append(VARS_END); return sb.toString(); } - - public static String serializeDataObject(String key, Data dat) - { + + public static String serializeDataObject(String key, Data dat) { // SCHEMA: |||value // (scalars are serialize by value, matrices by filename) StringBuilder sb = new StringBuilder(); @@ -941,8 +891,7 @@ public static String serializeDataObject(String key, Data dat) String value = null; String[] metaData = null; String[] listData = null; - switch( datatype ) - { + switch (datatype) { case SCALAR: ScalarObject so = (ScalarObject) dat; //name = so.getName(); @@ -953,16 +902,16 @@ public static String serializeDataObject(String key, Data dat) MetaDataFormat md = (MetaDataFormat) dat.getMetaData(); DataCharacteristics dc = md.getDataCharacteristics(); value = mo.getFileName(); - PartitionFormat partFormat = (mo.getPartitionFormat()!=null) ? new PartitionFormat( - mo.getPartitionFormat(),mo.getPartitionSize()) : PartitionFormat.NONE; + PartitionFormat partFormat = (mo.getPartitionFormat() != null) ? new PartitionFormat( + mo.getPartitionFormat(), mo.getPartitionSize()) : PartitionFormat.NONE; metaData = new String[10]; - metaData[0] = String.valueOf( dc.getRows() ); - metaData[1] = String.valueOf( dc.getCols() ); - metaData[2] = String.valueOf( dc.getBlocksize() ); - metaData[3] = String.valueOf( dc.getNonZeros() ); + metaData[0] = String.valueOf(dc.getRows()); + metaData[1] = String.valueOf(dc.getCols()); + metaData[2] = String.valueOf(dc.getBlocksize()); + metaData[3] = String.valueOf(dc.getNonZeros()); metaData[4] = md.getFileFormat().toString(); - metaData[5] = String.valueOf( partFormat ); - metaData[6] = String.valueOf( mo.getUpdateType() ); + metaData[5] = String.valueOf(partFormat); + metaData[6] = String.valueOf(mo.getUpdateType()); metaData[7] = String.valueOf(mo.isHDFSFileExists()); metaData[8] = String.valueOf(mo.isCleanupEnabled()); break; @@ -972,13 +921,16 @@ public static String serializeDataObject(String key, Data dat) MetaDataFormat md = (MetaDataFormat) dat.getMetaData(); DataCharacteristics dc = md.getDataCharacteristics(); value = fo.getFileName(); - metaData = new String[6]; + metaData = new String[7]; metaData[0] = String.valueOf(dc.getRows()); metaData[1] = String.valueOf(dc.getCols()); metaData[2] = String.valueOf(dc.getBlocksize()); metaData[3] = md.getFileFormat().toString(); metaData[4] = String.valueOf(fo.isHDFSFileExists()); metaData[5] = String.valueOf(fo.isCleanupEnabled()); + metaData[6] = fo.getColumnNames() == null + ? EMPTY + : serializeList(Arrays.asList(fo.getColumnNames()), ELEMENT_DELIM2); break; } case LIST: @@ -996,9 +948,9 @@ public static String serializeDataObject(String key, Data dat) } break; default: - throw new DMLRuntimeException("Unable to serialize datatype "+datatype); + throw new DMLRuntimeException("Unable to serialize datatype " + datatype); } - + //serialize data sb.append(name); sb.append(DATA_FIELD_DELIM); @@ -1007,8 +959,8 @@ public static String serializeDataObject(String key, Data dat) sb.append(valuetype); sb.append(DATA_FIELD_DELIM); sb.append(value); - if( metaData != null ) - for( int i=0; i inst, HashMap clsMap ) - { + private static String serializeInstructions(ArrayList inst, HashMap clsMap) { StringBuilder sb = new StringBuilder(); int count = 0; - for( Instruction linst : inst ) { + for (Instruction linst : inst) { //check that only cp instruction are transmitted - if( !( linst instanceof CPInstruction) ) - throw new DMLRuntimeException( NOT_SUPPORTED_SPARK_INSTRUCTION + " " +linst.getClass().getName()+"\n"+linst ); - + if (!(linst instanceof CPInstruction)) + throw new DMLRuntimeException(NOT_SUPPORTED_SPARK_INSTRUCTION + " " + linst.getClass().getName() + "\n" + linst); + //obtain serialized version of generated classes - if( linst instanceof SpoofCPInstruction ) { + if (linst instanceof SpoofCPInstruction) { Class cla = ((SpoofCPInstruction) linst).getOperatorClass(); clsMap.put(cla.getName(), CodegenUtils.getClassData(cla.getName())); } - - if( count > 0 ) - sb.append( ELEMENT_DELIM ); - - sb.append( checkAndReplaceLiterals( linst.toString() ) ); + + if (count > 0) + sb.append(ELEMENT_DELIM); + + sb.append(checkAndReplaceLiterals(linst.toString())); count++; } - + return sb.toString(); } - + /** * Replacement of internal delimiters occurring in literals of instructions * in order to ensure robustness of serialization and parsing. * (e.g. print( "a,b" ) would break the parsing of instruction that internally * are separated with a "," ) - * + * * @param instStr instruction string * @return instruction string with replacements */ - private static String checkAndReplaceLiterals( String instStr ) - { + private static String checkAndReplaceLiterals(String instStr) { String tmp = instStr; - + //1) check own delimiters (very unlikely due to special characters) - if( tmp.contains(COMPONENTS_DELIM) ) { + if (tmp.contains(COMPONENTS_DELIM)) { tmp = tmp.replaceAll(COMPONENTS_DELIM, "."); - LOG.warn("Replaced special literal character sequence "+COMPONENTS_DELIM+" with '.'"); + LOG.warn("Replaced special literal character sequence " + COMPONENTS_DELIM + " with '.'"); } - - if( tmp.contains(ELEMENT_DELIM) ) { + + if (tmp.contains(ELEMENT_DELIM)) { tmp = tmp.replaceAll(ELEMENT_DELIM, "."); - LOG.warn("Replaced special literal character sequence "+ELEMENT_DELIM+" with '.'"); + LOG.warn("Replaced special literal character sequence " + ELEMENT_DELIM + " with '.'"); } - - if( tmp.contains( LEVELIN ) ){ + + if (tmp.contains(LEVELIN)) { tmp = tmp.replaceAll(LEVELIN, "("); // '\\' required if LEVELIN='{' because regex - LOG.warn("Replaced special literal character sequence "+LEVELIN+" with '('"); + LOG.warn("Replaced special literal character sequence " + LEVELIN + " with '('"); } - if( tmp.contains(LEVELOUT) ){ + if (tmp.contains(LEVELOUT)) { tmp = tmp.replaceAll(LEVELOUT, ")"); - LOG.warn("Replaced special literal character sequence "+LEVELOUT+" with ')'"); + LOG.warn("Replaced special literal character sequence " + LEVELOUT + " with ')'"); } - + //NOTE: DATA_FIELD_DELIM and KEY_VALUE_DELIM not required //because those literals cannot occur in critical places. - + //2) check end tag of CDATA - if( tmp.contains(CDATA_END) ){ + if (tmp.contains(CDATA_END)) { tmp = tmp.replaceAll(CDATA_END, "."); //prevent XML parsing issues in job.xml - LOG.warn("Replaced special literal character sequence "+ CDATA_END +" with '.'"); + LOG.warn("Replaced special literal character sequence " + CDATA_END + " with '.'"); } - + return tmp; } - private static String serializeStringHashMap(HashMap vars) { + private static String serializeStringHashMap(HashMap vars) { return serializeList(vars.entrySet().stream().map(e -> - e.getKey()+KEY_VALUE_DELIM+e.getValue()).collect(Collectors.toList())); + e.getKey() + KEY_VALUE_DELIM + e.getValue()).collect(Collectors.toList())); } - public static String serializeResultVariables( List vars) { + public static String serializeResultVariables(List vars) { return serializeList(vars.stream().map(v -> v._isAccum ? - v._name+"+" : v._name).collect(Collectors.toList())); + v._name + "+" : v._name).collect(Collectors.toList())); } - + public static String serializeList(List elements) { return serializeList(elements, ELEMENT_DELIM); } - + public static String serializeList(List elements, String delim) { return StringUtils.join(elements, delim); } private static String serializeDataIdentifiers(List vars) { return serializeList(vars.stream().map(v -> - serializeDataIdentifier(v)).collect(Collectors.toList())); + serializeDataIdentifier(v)).collect(Collectors.toList())); } - private static String serializeDataIdentifier( DataIdentifier dat ) { + private static String serializeDataIdentifier(DataIdentifier dat) { // SCHEMA: || StringBuilder sb = new StringBuilder(); sb.append(dat.getName()); @@ -1136,25 +1086,25 @@ private static String serializeDataIdentifier( DataIdentifier dat ) { private static String rSerializeFunctionProgramBlocks(Program prog, HashSet cand, HashMap clsMap) { StringBuilder sb = new StringBuilder(); int count = 0; - for( String fkey : prog.getFunctionProgramBlocks().keySet() ) { - if( !cand.contains(fkey) ) //skip function not included in the parfor body + for (String fkey : prog.getFunctionProgramBlocks().keySet()) { + if (!cand.contains(fkey)) //skip function not included in the parfor body continue; - if( count>0 ) - sb.append( ELEMENT_DELIM ); - sb.append( fkey ); - sb.append( KEY_VALUE_DELIM ); + if (count > 0) + sb.append(ELEMENT_DELIM); + sb.append(fkey); + sb.append(KEY_VALUE_DELIM); FunctionProgramBlock fpb1 = prog.getFunctionProgramBlock(fkey, true); - sb.append( rSerializeProgramBlock(fpb1, clsMap) ); - if( prog.containsFunctionProgramBlock(fkey, false) ) { - sb.append( ELEMENT_DELIM ); - sb.append( fkey ); - sb.append( KEY_VALUE_DELIM ); + sb.append(rSerializeProgramBlock(fpb1, clsMap)); + if (prog.containsFunctionProgramBlock(fkey, false)) { + sb.append(ELEMENT_DELIM); + sb.append(fkey); + sb.append(KEY_VALUE_DELIM); FunctionProgramBlock fpb2 = prog.getFunctionProgramBlock(fkey, false); - if( OptTreeConverter.rContainsSparkInstruction(fpb2.getChildBlocks(), false) ) { + if (OptTreeConverter.rContainsSparkInstruction(fpb2.getChildBlocks(), false)) { Recompiler.recompileProgramBlockHierarchy2Forced( - fpb2.getChildBlocks(), -1, new HashSet<>(), ExecType.CP); + fpb2.getChildBlocks(), -1, new HashSet<>(), ExecType.CP); } - sb.append( rSerializeProgramBlock(fpb2, clsMap) ); + sb.append(rSerializeProgramBlock(fpb2, clsMap)); } count++; } @@ -1165,139 +1115,135 @@ private static String rSerializeFunctionProgramBlocks(Program prog, HashSet pbs, HashMap clsMap) { StringBuilder sb = new StringBuilder(); int count = 0; - for( ProgramBlock pb : pbs ) { - if( count>0 ) { - sb.append( ELEMENT_DELIM ); + for (ProgramBlock pb : pbs) { + if (count > 0) { + sb.append(ELEMENT_DELIM); } - sb.append( rSerializeProgramBlock(pb, clsMap) ); + sb.append(rSerializeProgramBlock(pb, clsMap)); count++; } return sb.toString(); } - private static String rSerializeProgramBlock( ProgramBlock pb, HashMap clsMap ) { + private static String rSerializeProgramBlock(ProgramBlock pb, HashMap clsMap) { StringBuilder sb = new StringBuilder(); - - boolean pbFOR = pb instanceof ForProgramBlock - && (!(pb instanceof ParForProgramBlock) || ParForProgramBlock.CONVERT_NESTED_REMOTE_PARFOR); - + + boolean pbFOR = pb instanceof ForProgramBlock + && (!(pb instanceof ParForProgramBlock) || ParForProgramBlock.CONVERT_NESTED_REMOTE_PARFOR); + //handle header - if( pb instanceof WhileProgramBlock ) + if (pb instanceof WhileProgramBlock) sb.append(PB_WHILE); - else if ( pbFOR ) + else if (pbFOR) sb.append(PB_FOR); - else if ( pb instanceof ParForProgramBlock ) + else if (pb instanceof ParForProgramBlock) sb.append(PB_PARFOR); - else if ( pb instanceof IfProgramBlock ) + else if (pb instanceof IfProgramBlock) sb.append(PB_IF); - else if ( pb instanceof FunctionProgramBlock ) + else if (pb instanceof FunctionProgramBlock) sb.append(PB_FC); else //all generic program blocks sb.append(PB_BEGIN); - + //handle body - if( pb instanceof WhileProgramBlock ) { + if (pb instanceof WhileProgramBlock) { WhileProgramBlock wpb = (WhileProgramBlock) pb; sb.append(INST_BEGIN); - sb.append( serializeInstructions( wpb.getPredicate(), clsMap ) ); + sb.append(serializeInstructions(wpb.getPredicate(), clsMap)); sb.append(INST_END); - sb.append( COMPONENTS_DELIM ); + sb.append(COMPONENTS_DELIM); sb.append(PBS_BEGIN); - sb.append( rSerializeProgramBlocks( wpb.getChildBlocks(), clsMap) ); + sb.append(rSerializeProgramBlocks(wpb.getChildBlocks(), clsMap)); sb.append(PBS_END); - } - else if ( pbFOR ) { // might catch parfor too - ForProgramBlock fpb = (ForProgramBlock) pb; - sb.append( fpb.getIterVar() ); - sb.append( COMPONENTS_DELIM ); + } else if (pbFOR) { // might catch parfor too + ForProgramBlock fpb = (ForProgramBlock) pb; + sb.append(fpb.getIterVar()); + sb.append(COMPONENTS_DELIM); sb.append(INST_BEGIN); - sb.append( serializeInstructions( fpb.getFromInstructions(), clsMap ) ); + sb.append(serializeInstructions(fpb.getFromInstructions(), clsMap)); sb.append(INST_END); - sb.append( COMPONENTS_DELIM ); + sb.append(COMPONENTS_DELIM); sb.append(INST_BEGIN); - sb.append( serializeInstructions(fpb.getToInstructions(), clsMap) ); + sb.append(serializeInstructions(fpb.getToInstructions(), clsMap)); sb.append(INST_END); - sb.append( COMPONENTS_DELIM ); + sb.append(COMPONENTS_DELIM); sb.append(INST_BEGIN); - sb.append( serializeInstructions(fpb.getIncrementInstructions(), clsMap) ); + sb.append(serializeInstructions(fpb.getIncrementInstructions(), clsMap)); sb.append(INST_END); - sb.append( COMPONENTS_DELIM ); + sb.append(COMPONENTS_DELIM); sb.append(PBS_BEGIN); - sb.append( rSerializeProgramBlocks(fpb.getChildBlocks(), clsMap) ); + sb.append(rSerializeProgramBlocks(fpb.getChildBlocks(), clsMap)); sb.append(PBS_END); - } - else if ( pb instanceof ParForProgramBlock ) { - ParForProgramBlock pfpb = (ParForProgramBlock) pb; - + } else if (pb instanceof ParForProgramBlock) { + ParForProgramBlock pfpb = (ParForProgramBlock) pb; + //check for nested remote ParFOR - if( PExecMode.valueOf( pfpb.getParForParams().get( ParForStatementBlock.EXEC_MODE )) == PExecMode.REMOTE_SPARK ) - throw new DMLRuntimeException( NOT_SUPPORTED_SPARK_PARFOR ); - - sb.append( pfpb.getIterVar() ); - sb.append( COMPONENTS_DELIM ); - sb.append( serializeResultVariables( pfpb.getResultVariables()) ); - sb.append( COMPONENTS_DELIM ); - sb.append( serializeStringHashMap( pfpb.getParForParams()) ); //parameters of nested parfor - sb.append( COMPONENTS_DELIM ); + if (PExecMode.valueOf(pfpb.getParForParams().get(ParForStatementBlock.EXEC_MODE)) == PExecMode.REMOTE_SPARK) + throw new DMLRuntimeException(NOT_SUPPORTED_SPARK_PARFOR); + + sb.append(pfpb.getIterVar()); + sb.append(COMPONENTS_DELIM); + sb.append(serializeResultVariables(pfpb.getResultVariables())); + sb.append(COMPONENTS_DELIM); + sb.append(serializeStringHashMap(pfpb.getParForParams())); //parameters of nested parfor + sb.append(COMPONENTS_DELIM); sb.append(INST_BEGIN); - sb.append( serializeInstructions(pfpb.getFromInstructions(), clsMap) ); + sb.append(serializeInstructions(pfpb.getFromInstructions(), clsMap)); sb.append(INST_END); - sb.append( COMPONENTS_DELIM ); + sb.append(COMPONENTS_DELIM); sb.append(INST_BEGIN); - sb.append( serializeInstructions(pfpb.getToInstructions(), clsMap) ); + sb.append(serializeInstructions(pfpb.getToInstructions(), clsMap)); sb.append(INST_END); - sb.append( COMPONENTS_DELIM ); + sb.append(COMPONENTS_DELIM); sb.append(INST_BEGIN); - sb.append( serializeInstructions(pfpb.getIncrementInstructions(), clsMap) ); + sb.append(serializeInstructions(pfpb.getIncrementInstructions(), clsMap)); sb.append(INST_END); - sb.append( COMPONENTS_DELIM ); + sb.append(COMPONENTS_DELIM); sb.append(PBS_BEGIN); - sb.append( rSerializeProgramBlocks( pfpb.getChildBlocks(), clsMap ) ); + sb.append(rSerializeProgramBlocks(pfpb.getChildBlocks(), clsMap)); sb.append(PBS_END); - } - else if ( pb instanceof IfProgramBlock ) { + } else if (pb instanceof IfProgramBlock) { IfProgramBlock ipb = (IfProgramBlock) pb; sb.append(INST_BEGIN); - sb.append( serializeInstructions(ipb.getPredicate(), clsMap) ); + sb.append(serializeInstructions(ipb.getPredicate(), clsMap)); sb.append(INST_END); - sb.append( COMPONENTS_DELIM ); + sb.append(COMPONENTS_DELIM); sb.append(PBS_BEGIN); - sb.append( rSerializeProgramBlocks(ipb.getChildBlocksIfBody(), clsMap) ); + sb.append(rSerializeProgramBlocks(ipb.getChildBlocksIfBody(), clsMap)); sb.append(PBS_END); - sb.append( COMPONENTS_DELIM ); + sb.append(COMPONENTS_DELIM); sb.append(PBS_BEGIN); - sb.append( rSerializeProgramBlocks(ipb.getChildBlocksElseBody(), clsMap) ); + sb.append(rSerializeProgramBlocks(ipb.getChildBlocksElseBody(), clsMap)); sb.append(PBS_END); - } - else if( pb instanceof FunctionProgramBlock ) { + } else if (pb instanceof FunctionProgramBlock) { FunctionProgramBlock fpb = (FunctionProgramBlock) pb; - sb.append( serializeDataIdentifiers( fpb.getInputParams() ) ); - sb.append( COMPONENTS_DELIM ); - sb.append( serializeDataIdentifiers( fpb.getOutputParams() ) ); - sb.append( COMPONENTS_DELIM ); + sb.append(serializeDataIdentifiers(fpb.getInputParams())); + sb.append(COMPONENTS_DELIM); + sb.append(serializeDataIdentifiers(fpb.getOutputParams())); + sb.append(COMPONENTS_DELIM); sb.append(PBS_BEGIN); - sb.append( rSerializeProgramBlocks(fpb.getChildBlocks(), clsMap) ); + sb.append(rSerializeProgramBlocks(fpb.getChildBlocks(), clsMap)); sb.append(PBS_END); - sb.append( COMPONENTS_DELIM ); - } - else if( pb instanceof BasicProgramBlock ) { + sb.append(COMPONENTS_DELIM); + } else if (pb instanceof BasicProgramBlock) { BasicProgramBlock bpb = (BasicProgramBlock) pb; sb.append(INST_BEGIN); - sb.append( serializeInstructions( - bpb.getInstructions(), clsMap) ); + sb.append(serializeInstructions( + bpb.getInstructions(), clsMap)); sb.append(INST_END); } - + //handle end sb.append(PB_END); - + return sb.toString(); } - + //////////////////////////////// // PARSING - //////////////////////////////// + + /// ///////////////////////////// public static SparkPSBody parseSparkPSBody(String in, int id) { SparkPSBody body = new SparkPSBody(); @@ -1333,59 +1279,59 @@ public static SparkPSBody parseSparkPSBody(String in, int id) { return body; } - public static ParForBody parseParForBody( String in, int id ) { + public static ParForBody parseParForBody(String in, int id) { return parseParForBody(in, id, false); } - - public static ParForBody parseParForBody( String in, int id, boolean inSpark ) { + + public static ParForBody parseParForBody(String in, int id, boolean inSpark) { ParForBody body = new ParForBody(); - + //header elimination String tmpin = in.replaceAll(NEWLINE, ""); //normalization - tmpin = tmpin.substring(PARFORBODY_BEGIN.length(),tmpin.length()-PARFORBODY_END.length()); //remove start/end + tmpin = tmpin.substring(PARFORBODY_BEGIN.length(), tmpin.length() - PARFORBODY_END.length()); //remove start/end HierarchyAwareStringTokenizer st = new HierarchyAwareStringTokenizer(tmpin, COMPONENTS_DELIM); - + //handle DMLScript UUID (NOTE: set directly in DMLScript) //(master UUID is used for all nodes (in order to simply cleanup)) - DMLScript.setUUID( st.nextToken() ); - + DMLScript.setUUID(st.nextToken()); + //handle DML config (NOTE: set directly in ConfigurationManager) String confStr = st.nextToken(); JobConf job = ConfigurationManager.getCachedJobConf(); - if( !InfrastructureAnalyzer.isLocalMode(job) ) { + if (!InfrastructureAnalyzer.isLocalMode(job)) { handleDMLConfig(confStr); } - + //handle additional configs String aconfs = st.nextToken(); - if( !inSpark ) - parseAndSetAdditionalConfigurations( aconfs ); - + if (!inSpark) + parseAndSetAdditionalConfigurations(aconfs); + //handle program String progStr = st.nextToken(); - Program prog = parseProgram( progStr, id ); - + Program prog = parseProgram(progStr, id); + //handle result variable names String rvarStr = st.nextToken(); ArrayList rvars = parseResultVariables(rvarStr); body.setResultVariables(rvars); - + //handle execution context String ecStr = st.nextToken(); - ExecutionContext ec = parseExecutionContext( ecStr, prog ); - + ExecutionContext ec = parseExecutionContext(ecStr, prog); + //handle program blocks String spbs = st.nextToken(); ArrayList pbs = rParseProgramBlocks(spbs, prog, id); - - body.setChildBlocks( pbs ); - body.setEc( ec ); - + + body.setChildBlocks(pbs); + body.setEc(ec); + return body; } private static void handleDMLConfig(String confStr) { - if(confStr != null && !confStr.trim().isEmpty()) { + if (confStr != null && !confStr.trim().isEmpty()) { DMLConfig dmlconf = DMLConfig.parseDMLConfig(confStr); CompilerConfig cconf = OptimizerUtils.constructCompilerConfig(dmlconf); ConfigurationManager.setLocalConfig(dmlconf); @@ -1393,8 +1339,8 @@ private static void handleDMLConfig(String confStr) { } } - public static Program parseProgram( String in, int id ) { - String lin = in.substring( PROG_BEGIN.length(),in.length()- PROG_END.length()).trim(); + public static Program parseProgram(String in, int id) { + String lin = in.substring(PROG_BEGIN.length(), in.length() - PROG_END.length()).trim(); Program prog = new Program(new DMLProgram()); parseFunctionProgramBlocks(lin, prog, id); return prog; @@ -1402,148 +1348,147 @@ public static Program parseProgram( String in, int id ) { private static LocalVariableMap parseVariables(String in) { LocalVariableMap ret = null; - if( in.length()> VARS_BEGIN.length() + VARS_END.length()) { - String varStr = in.substring( VARS_BEGIN.length(),in.length() - VARS_END.length()).trim(); + if (in.length() > VARS_BEGIN.length() + VARS_END.length()) { + String varStr = in.substring(VARS_BEGIN.length(), in.length() - VARS_END.length()).trim(); ret = LocalVariableMap.deserialize(varStr); - } - else { //empty input symbol table + } else { //empty input symbol table ret = new LocalVariableMap(); } return ret; } - - private static HashMap parseFunctionProgramBlocks( String in, Program prog, int id ) { - HashMap ret = new HashMap<>(); - HierarchyAwareStringTokenizer st = new HierarchyAwareStringTokenizer( in, ELEMENT_DELIM ); - while( st.hasMoreTokens() ) { - String lvar = st.nextToken(); //with ID = CP_CHILD_THREAD+id for current use + + private static HashMap parseFunctionProgramBlocks(String in, Program prog, int id) { + HashMap ret = new HashMap<>(); + HierarchyAwareStringTokenizer st = new HierarchyAwareStringTokenizer(in, ELEMENT_DELIM); + while (st.hasMoreTokens()) { + String lvar = st.nextToken(); //with ID = CP_CHILD_THREAD+id for current use //put first copy into prog (for direct use) - int index = lvar.indexOf( KEY_VALUE_DELIM ); + int index = lvar.indexOf(KEY_VALUE_DELIM); String fkey = lvar.substring(0, index); String tmp = lvar.substring(index + 1); boolean opt = !prog.containsFunctionProgramBlock(fkey, true); prog.addFunctionProgramBlock(fkey, - (FunctionProgramBlock)rParseProgramBlock(tmp, prog, id), opt); + (FunctionProgramBlock) rParseProgramBlock(tmp, prog, id), opt); } return ret; } private static ArrayList rParseProgramBlocks(String in, Program prog, int id) { ArrayList pbs = new ArrayList<>(); - String tmpdata = in.substring(PBS_BEGIN.length(),in.length()- PBS_END.length()); + String tmpdata = in.substring(PBS_BEGIN.length(), in.length() - PBS_END.length()); HierarchyAwareStringTokenizer st = new HierarchyAwareStringTokenizer(tmpdata, ELEMENT_DELIM); - while( st.hasMoreTokens() ) - pbs.add( rParseProgramBlock( st.nextToken(), prog, id ) ); + while (st.hasMoreTokens()) + pbs.add(rParseProgramBlock(st.nextToken(), prog, id)); return pbs; } - private static ProgramBlock rParseProgramBlock( String in, Program prog, int id ) { + private static ProgramBlock rParseProgramBlock(String in, Program prog, int id) { ProgramBlock pb = null; - if( in.startsWith(PB_WHILE) ) - pb = rParseWhileProgramBlock( in, prog, id ); - else if ( in.startsWith(PB_FOR) ) - pb = rParseForProgramBlock( in, prog, id ); - else if ( in.startsWith(PB_PARFOR) ) - pb = rParseParForProgramBlock( in, prog, id ); - else if ( in.startsWith(PB_IF) ) - pb = rParseIfProgramBlock( in, prog, id ); - else if ( in.startsWith(PB_FC) ) - pb = rParseFunctionProgramBlock( in, prog, id ); - else if ( in.startsWith(PB_BEGIN) ) - pb = rParseGenericProgramBlock( in, prog, id ); - else - throw new DMLRuntimeException( NOT_SUPPORTED_PB+" "+in ); + if (in.startsWith(PB_WHILE)) + pb = rParseWhileProgramBlock(in, prog, id); + else if (in.startsWith(PB_FOR)) + pb = rParseForProgramBlock(in, prog, id); + else if (in.startsWith(PB_PARFOR)) + pb = rParseParForProgramBlock(in, prog, id); + else if (in.startsWith(PB_IF)) + pb = rParseIfProgramBlock(in, prog, id); + else if (in.startsWith(PB_FC)) + pb = rParseFunctionProgramBlock(in, prog, id); + else if (in.startsWith(PB_BEGIN)) + pb = rParseGenericProgramBlock(in, prog, id); + else + throw new DMLRuntimeException(NOT_SUPPORTED_PB + " " + in); return pb; } - private static WhileProgramBlock rParseWhileProgramBlock( String in, Program prog, int id ) { - String lin = in.substring( PB_WHILE.length(),in.length()- PB_END.length()); + private static WhileProgramBlock rParseWhileProgramBlock(String in, Program prog, int id) { + String lin = in.substring(PB_WHILE.length(), in.length() - PB_END.length()); HierarchyAwareStringTokenizer st = new HierarchyAwareStringTokenizer(lin, COMPONENTS_DELIM); - + //predicate instructions - ArrayList inst = parseInstructions(st.nextToken(),id); - + ArrayList inst = parseInstructions(st.nextToken(), id); + //program blocks ArrayList pbs = rParseProgramBlocks(st.nextToken(), prog, id); - - WhileProgramBlock wpb = new WhileProgramBlock(prog,inst); + + WhileProgramBlock wpb = new WhileProgramBlock(prog, inst); wpb.setChildBlocks(pbs); return wpb; } - private static ForProgramBlock rParseForProgramBlock( String in, Program prog, int id ) { - String lin = in.substring( PB_FOR.length(),in.length()- PB_END.length()); + private static ForProgramBlock rParseForProgramBlock(String in, Program prog, int id) { + String lin = in.substring(PB_FOR.length(), in.length() - PB_END.length()); HierarchyAwareStringTokenizer st = new HierarchyAwareStringTokenizer(lin, COMPONENTS_DELIM); - + //inputs String iterVar = st.nextToken(); - + //instructions - ArrayList from = parseInstructions(st.nextToken(),id); - ArrayList to = parseInstructions(st.nextToken(),id); - ArrayList incr = parseInstructions(st.nextToken(),id); - + ArrayList from = parseInstructions(st.nextToken(), id); + ArrayList to = parseInstructions(st.nextToken(), id); + ArrayList incr = parseInstructions(st.nextToken(), id); + //program blocks ArrayList pbs = rParseProgramBlocks(st.nextToken(), prog, id); - + ForProgramBlock fpb = new ForProgramBlock(prog, iterVar); fpb.setFromInstructions(from); fpb.setToInstructions(to); fpb.setIncrementInstructions(incr); fpb.setChildBlocks(pbs); - + return fpb; } - private static ParForProgramBlock rParseParForProgramBlock( String in, Program prog, int id ) { - String lin = in.substring( PB_PARFOR.length(),in.length()- PB_END.length()); + private static ParForProgramBlock rParseParForProgramBlock(String in, Program prog, int id) { + String lin = in.substring(PB_PARFOR.length(), in.length() - PB_END.length()); HierarchyAwareStringTokenizer st = new HierarchyAwareStringTokenizer(lin, COMPONENTS_DELIM); - + //inputs String iterVar = st.nextToken(); ArrayList resultVars = parseResultVariables(st.nextToken()); - HashMap params = parseStringHashMap(st.nextToken()); - + HashMap params = parseStringHashMap(st.nextToken()); + //instructions ArrayList from = parseInstructions(st.nextToken(), 0); ArrayList to = parseInstructions(st.nextToken(), 0); ArrayList incr = parseInstructions(st.nextToken(), 0); - + //program blocks //reset id to preinit state, replaced during exec - ArrayList pbs = rParseProgramBlocks(st.nextToken(), prog, 0); - + ArrayList pbs = rParseProgramBlocks(st.nextToken(), prog, 0); + ParForProgramBlock pfpb = new ParForProgramBlock(id, prog, iterVar, params, resultVars); pfpb.disableOptimization(); //already done in top-level parfor pfpb.setFromInstructions(from); pfpb.setToInstructions(to); pfpb.setIncrementInstructions(incr); pfpb.setChildBlocks(pbs); - + return pfpb; } - private static IfProgramBlock rParseIfProgramBlock( String in, Program prog, int id ) { - String lin = in.substring( PB_IF.length(),in.length()- PB_END.length()); + private static IfProgramBlock rParseIfProgramBlock(String in, Program prog, int id) { + String lin = in.substring(PB_IF.length(), in.length() - PB_END.length()); HierarchyAwareStringTokenizer st = new HierarchyAwareStringTokenizer(lin, COMPONENTS_DELIM); - + //predicate instructions - ArrayList inst = parseInstructions(st.nextToken(),id); - + ArrayList inst = parseInstructions(st.nextToken(), id); + //program blocks: if and else ArrayList pbs1 = rParseProgramBlocks(st.nextToken(), prog, id); ArrayList pbs2 = rParseProgramBlocks(st.nextToken(), prog, id); - - IfProgramBlock ipb = new IfProgramBlock(prog,inst); + + IfProgramBlock ipb = new IfProgramBlock(prog, inst); ipb.setChildBlocksIfBody(pbs1); ipb.setChildBlocksElseBody(pbs2); - + return ipb; } - private static FunctionProgramBlock rParseFunctionProgramBlock( String in, Program prog, int id ) { - String lin = in.substring( PB_FC.length(),in.length()- PB_END.length()); + private static FunctionProgramBlock rParseFunctionProgramBlock(String in, Program prog, int id) { + String lin = in.substring(PB_FC.length(), in.length() - PB_END.length()); HierarchyAwareStringTokenizer st = new HierarchyAwareStringTokenizer(lin, COMPONENTS_DELIM); - + //inputs and outputs ArrayList dat1 = parseDataIdentifiers(st.nextToken()); ArrayList dat2 = parseDataIdentifiers(st.nextToken()); @@ -1555,59 +1500,58 @@ private static FunctionProgramBlock rParseFunctionProgramBlock( String in, Progr ArrayList tmp2 = new ArrayList<>(dat2); FunctionProgramBlock fpb = new FunctionProgramBlock(prog, tmp1, tmp2); fpb.setChildBlocks(pbs); - + return fpb; } - private static ProgramBlock rParseGenericProgramBlock( String in, Program prog, int id ) { - String lin = in.substring( PB_BEGIN.length(),in.length()- PB_END.length()); - StringTokenizer st = new StringTokenizer(lin,COMPONENTS_DELIM); + private static ProgramBlock rParseGenericProgramBlock(String in, Program prog, int id) { + String lin = in.substring(PB_BEGIN.length(), in.length() - PB_END.length()); + StringTokenizer st = new StringTokenizer(lin, COMPONENTS_DELIM); BasicProgramBlock pb = new BasicProgramBlock(prog); - pb.setInstructions(parseInstructions(st.nextToken(),id)); + pb.setInstructions(parseInstructions(st.nextToken(), id)); return pb; } - private static ArrayList parseInstructions( String in, int id ) { + private static ArrayList parseInstructions(String in, int id) { ArrayList insts = new ArrayList<>(); - String lin = in.substring( INST_BEGIN.length(),in.length()- INST_END.length()); + String lin = in.substring(INST_BEGIN.length(), in.length() - INST_END.length()); StringTokenizer st = new StringTokenizer(lin, ELEMENT_DELIM); - while(st.hasMoreTokens()) { + while (st.hasMoreTokens()) { //Note that at this point only CP instructions and External function instruction can occur String instStr = st.nextToken(); try { Instruction tmpinst = CPInstructionParser.parseSingleInstruction(instStr); - tmpinst = saveReplaceThreadID(tmpinst, Lop.CP_ROOT_THREAD_ID, Lop.CP_CHILD_THREAD+id ); - insts.add( tmpinst ); - } - catch(Exception ex) { + tmpinst = saveReplaceThreadID(tmpinst, Lop.CP_ROOT_THREAD_ID, Lop.CP_CHILD_THREAD + id); + insts.add(tmpinst); + } catch (Exception ex) { throw new DMLRuntimeException("Failed to parse instruction: " + instStr, ex); } } return insts; } - + private static ArrayList parseResultVariables(String in) { ArrayList ret = new ArrayList<>(); - for(String var : parseStringArrayList(in)) { + for (String var : parseStringArrayList(in)) { boolean accum = var.endsWith("+"); - ret.add(new ResultVar(accum ? var.substring(0, var.length()-1) : var, accum)); + ret.add(new ResultVar(accum ? var.substring(0, var.length() - 1) : var, accum)); } return ret; } - private static HashMap parseStringHashMap( String in ) { - HashMap vars = new HashMap<>(); - StringTokenizer st = new StringTokenizer(in,ELEMENT_DELIM); - while( st.hasMoreTokens() ) { + private static HashMap parseStringHashMap(String in) { + HashMap vars = new HashMap<>(); + StringTokenizer st = new StringTokenizer(in, ELEMENT_DELIM); + while (st.hasMoreTokens()) { String lin = st.nextToken(); - int index = lin.indexOf( KEY_VALUE_DELIM ); + int index = lin.indexOf(KEY_VALUE_DELIM); String tmp1 = lin.substring(0, index); String tmp2 = lin.substring(index + 1); vars.put(tmp1, tmp2); } return vars; } - + private static ArrayList parseStringArrayList(String in) { return parseStringArrayList(in, ELEMENT_DELIM); } @@ -1615,73 +1559,80 @@ private static ArrayList parseStringArrayList(String in) { private static ArrayList parseStringArrayList(String in, String delim) { StringTokenizer st = new StringTokenizer(in, delim); ArrayList vars = new ArrayList<>(st.countTokens()); - while( st.hasMoreTokens() ) + while (st.hasMoreTokens()) vars.add(st.nextToken()); return vars; } - private static ArrayList parseDataIdentifiers( String in ) { + private static ArrayList parseDataIdentifiers(String in) { ArrayList vars = new ArrayList<>(); StringTokenizer st = new StringTokenizer(in, ELEMENT_DELIM); - while( st.hasMoreTokens() ) { + while (st.hasMoreTokens()) { String tmp = st.nextToken(); - DataIdentifier dat = parseDataIdentifier( tmp ); + DataIdentifier dat = parseDataIdentifier(tmp); vars.add(dat); } return vars; } - private static DataIdentifier parseDataIdentifier( String in ) { + private static DataIdentifier parseDataIdentifier(String in) { StringTokenizer st = new StringTokenizer(in, DATA_FIELD_DELIM); DataIdentifier dat = new DataIdentifier(st.nextToken()); dat.setDataType(DataType.valueOf(st.nextToken())); dat.setValueType(ValueType.valueOf(st.nextToken())); return dat; } - + /** * NOTE: MRJobConfiguration cannot be used for the general case because program blocks and * related symbol tables can be hierarchically structured. - * + * * @param in data object as string * @return array of objects */ public static Object[] parseDataObject(String in) { Object[] ret = new Object[2]; - - StringTokenizer st = new StringTokenizer(in, DATA_FIELD_DELIM ); + + StringTokenizer st = new StringTokenizer(in, DATA_FIELD_DELIM); String name = st.nextToken(); - DataType datatype = DataType.valueOf( st.nextToken() ); - ValueType valuetype = ValueType.valueOf( st.nextToken() ); + DataType datatype = DataType.valueOf(st.nextToken()); + ValueType valuetype = ValueType.valueOf(st.nextToken()); String valString = st.hasMoreTokens() ? st.nextToken() : ""; Data dat = null; - switch( datatype ) - { + switch (datatype) { case SCALAR: { - switch ( valuetype ) { - case INT64: dat = new IntObject(Long.parseLong(valString)); break; - case FP64: dat = new DoubleObject(Double.parseDouble(valString)); break; - case BOOLEAN: dat = new BooleanObject(Boolean.parseBoolean(valString)); break; - case STRING: dat = new StringObject(valString); break; + switch (valuetype) { + case INT64: + dat = new IntObject(Long.parseLong(valString)); + break; + case FP64: + dat = new DoubleObject(Double.parseDouble(valString)); + break; + case BOOLEAN: + dat = new BooleanObject(Boolean.parseBoolean(valString)); + break; + case STRING: + dat = new StringObject(valString); + break; default: - throw new DMLRuntimeException("Unable to parse valuetype "+valuetype); + throw new DMLRuntimeException("Unable to parse valuetype " + valuetype); } break; } case MATRIX: { - MatrixObject mo = new MatrixObject(valuetype,valString); - long rows = Long.parseLong( st.nextToken() ); - long cols = Long.parseLong( st.nextToken() ); - int blen = Integer.parseInt( st.nextToken() ); - long nnz = Long.parseLong( st.nextToken() ); + MatrixObject mo = new MatrixObject(valuetype, valString); + long rows = Long.parseLong(st.nextToken()); + long cols = Long.parseLong(st.nextToken()); + int blen = Integer.parseInt(st.nextToken()); + long nnz = Long.parseLong(st.nextToken()); FileFormat fmt = FileFormat.safeValueOf(st.nextToken()); - PartitionFormat partFormat = PartitionFormat.valueOf( st.nextToken() ); - UpdateType inplace = UpdateType.valueOf( st.nextToken() ); + PartitionFormat partFormat = PartitionFormat.valueOf(st.nextToken()); + UpdateType inplace = UpdateType.valueOf(st.nextToken()); MatrixCharacteristics mc = new MatrixCharacteristics(rows, cols, blen, nnz); MetaDataFormat md = new MetaDataFormat(mc, fmt); - mo.setMetaData( md ); - if( partFormat._dpf != PDataPartitionFormat.NONE ) - mo.setPartitioned( partFormat._dpf, partFormat._N ); + mo.setMetaData(md); + if (partFormat._dpf != PDataPartitionFormat.NONE) + mo.setPartitioned(partFormat._dpf, partFormat._N); mo.setUpdateType(inplace); mo.setHDFSFileExists(Boolean.valueOf(st.nextToken())); mo.enableCleanup(Boolean.valueOf(st.nextToken())); @@ -1696,9 +1647,22 @@ public static Object[] parseDataObject(String in) { FileFormat fmt = FileFormat.safeValueOf(st.nextToken()); MatrixCharacteristics mc = new MatrixCharacteristics(rows, cols, blen, -1); MetaDataFormat md = new MetaDataFormat(mc, fmt); - mo.setMetaData( md ); + mo.setMetaData(md); mo.setHDFSFileExists(Boolean.valueOf(st.nextToken())); mo.enableCleanup(Boolean.valueOf(st.nextToken())); + if(st.hasMoreTokens()) { + String namesString = st.nextToken(); + + if(!EMPTY.equals(namesString)) { + List names = + parseStringArrayList( + namesString, + ELEMENT_DELIM2); + + mo.setColumnNames( + names.toArray(new String[0])); + } + } dat = mo; break; } @@ -1706,7 +1670,7 @@ public static Object[] parseDataObject(String in) { int size = Integer.parseInt(st.nextToken()); String namesStr = st.nextToken(); List names = namesStr.equals(EMPTY) ? null : - parseStringArrayList(namesStr, ELEMENT_DELIM2); + parseStringArrayList(namesStr, ELEMENT_DELIM2); List data = new ArrayList<>(size); st.nextToken(LIST_ELEMENT_DELIM); for (int i = 0; i < size; i++) { @@ -1717,9 +1681,9 @@ public static Object[] parseDataObject(String in) { dat = new ListObject(data, names); break; default: - throw new DMLRuntimeException("Unable to parse datatype "+datatype); + throw new DMLRuntimeException("Unable to parse datatype " + datatype); } - + ret[0] = name; ret[1] = dat; return ret; @@ -1727,15 +1691,15 @@ public static Object[] parseDataObject(String in) { private static ExecutionContext parseExecutionContext(String in, Program prog) { ExecutionContext ec = null; - String lin = in.substring(EC_BEGIN.length(),in.length()- EC_END.length()).trim(); - if( !lin.equals( EMPTY ) ) { + String lin = in.substring(EC_BEGIN.length(), in.length() - EC_END.length()).trim(); + if (!lin.equals(EMPTY)) { LocalVariableMap vars = parseVariables(lin); - ec = ExecutionContextFactory.createContext( false, prog ); + ec = ExecutionContextFactory.createContext(false, prog); ec.setVariables(vars); } return ec; } - + private static void parseAndSetAdditionalConfigurations(String conf) { String[] statsFlag = conf.split("="); DMLScript.STATISTICS = Boolean.parseBoolean(statsFlag[1]); @@ -1743,52 +1707,52 @@ private static void parseAndSetAdditionalConfigurations(String conf) { ////////// // CUSTOM SAFE LITERAL REPLACEMENT - - + + /** * In-place replacement of thread ids in filenames, functions names etc - * - * @param inst instruction - * @param pattern ? + * + * @param inst instruction + * @param pattern ? * @param replacement string replacement * @return instruction */ - private static Instruction saveReplaceThreadID( Instruction inst, String pattern, String replacement ) { - if ( inst instanceof VariableCPInstruction //createvar, setfilename - || inst instanceof EvalNaryCPInstruction ) { + private static Instruction saveReplaceThreadID(Instruction inst, String pattern, String replacement) { + if (inst instanceof VariableCPInstruction //createvar, setfilename + || inst instanceof EvalNaryCPInstruction) { //update in-memory representation inst.updateInstructionThreadID(pattern, replacement); } //NOTE> //Rand, seq in CP not required return inst; } - + public static String saveReplaceFilenameThreadID(String fname, String pattern, String replace) { //save replace necessary in order to account for the possibility that read variables have our prefix in the absolute path //replace the last match only, because (1) we have at most one _t0 and (2) always concatenated to the end. int pos = fname.lastIndexOf(pattern); - return ( pos < 0 ) ? fname : fname.substring(0, pos) - + replace + fname.substring(pos+pattern.length()); + return (pos < 0) ? fname : fname.substring(0, pos) + + replace + fname.substring(pos + pattern.length()); } - - + + ////////// // CUSTOM HIERARCHICAL TOKENIZER - - + + /** * Custom StringTokenizer for splitting strings of hierarchies. The basic idea is to * search for delim-Strings on the same hierarchy level, while delims of lower hierarchy - * levels are skipped. - * + * levels are skipped. + * */ private static class HierarchyAwareStringTokenizer //extends StringTokenizer { private String _str = null; private String _del = null; - private int _off = -1; - - public HierarchyAwareStringTokenizer( String in, String delim ) { + private int _off = -1; + + public HierarchyAwareStringTokenizer(String in, String delim) { //super(in); _str = in; _del = delim; @@ -1802,48 +1766,46 @@ public boolean hasMoreTokens() { public String nextToken() { int nextDelim = determineNextSameLevelIndexOf(_str, _del); String token = null; - if(nextDelim < 0) { + if (nextDelim < 0) { nextDelim = _str.length(); _off = 0; } - token = _str.substring(0,nextDelim); - _str = _str.substring( nextDelim + _off ); + token = _str.substring(0, nextDelim); + _str = _str.substring(nextDelim + _off); return token; } - - private static int determineNextSameLevelIndexOf( String data, String pattern ) - { + + private static int determineNextSameLevelIndexOf(String data, String pattern) { String tmpdata = data; - int index = 0; - int count = 0; - int off=0,i1,i2,i3,min; - - while(true) { + int index = 0; + int count = 0; + int off = 0, i1, i2, i3, min; + + while (true) { i1 = tmpdata.indexOf(pattern); i2 = tmpdata.indexOf(LEVELIN); i3 = tmpdata.indexOf(LEVELOUT); - - if( i1 < 0 ) return i1; //no pattern found at all - + + if (i1 < 0) return i1; //no pattern found at all + min = i1; //min >= 0 by definition - if( i2 >= 0 ) min = Math.min(min, i2); - if( i3 >= 0 ) min = Math.min(min, i3); - + if (i2 >= 0) min = Math.min(min, i2); + if (i3 >= 0) min = Math.min(min, i3); + //stack maintenance - if( i1 == min && count == 0 ) - return index+i1; - else if( i2 == min ) { + if (i1 == min && count == 0) + return index + i1; + else if (i2 == min) { count++; off = LEVELIN.length(); - } - else if( i3 == min ) { + } else if (i3 == min) { count--; off = LEVELOUT.length(); } - + //prune investigated string - index += min+off; - tmpdata = tmpdata.substring(min+off); + index += min + off; + tmpdata = tmpdata.substring(min + off); } } } diff --git a/src/test/java/org/apache/sysds/test/component/paramserv/SerializationTest.java b/src/test/java/org/apache/sysds/test/component/paramserv/SerializationTest.java index 0fd172dfda5..54e4f25bf22 100644 --- a/src/test/java/org/apache/sysds/test/component/paramserv/SerializationTest.java +++ b/src/test/java/org/apache/sysds/test/component/paramserv/SerializationTest.java @@ -27,6 +27,10 @@ import java.util.Arrays; import java.util.Collection; +import org.apache.sysds.common.Types; +import org.apache.sysds.runtime.controlprogram.caching.FrameObject; +import org.apache.sysds.runtime.meta.MatrixCharacteristics; +import org.apache.sysds.runtime.meta.MetaDataFormat; import org.junit.Assert; import org.junit.Test; import org.apache.sysds.runtime.controlprogram.caching.MatrixObject; @@ -44,7 +48,7 @@ public class SerializationTest { @Parameterized.Parameters public static Collection named() { - return Arrays.asList(new Object[][] {{ 0 }, { 1 }}); + return Arrays.asList(new Object[][]{{0}, {1}}); } public SerializationTest(Integer named) { @@ -60,7 +64,7 @@ public void serializeListObject() { ListObject lo; if (_named == 1) - lo = new ListObject(Arrays.asList(mo1, lot, io), Arrays.asList("e1", "e2", "e3")); + lo = new ListObject(Arrays.asList(mo1, lot, io), Arrays.asList("e1", "e2", "e3")); else lo = new ListObject(Arrays.asList(mo1, lot, io)); @@ -77,10 +81,9 @@ public void serializeListObject() { ByteArrayInputStream bis = new ByteArrayInputStream(loBytes); ObjectInput in = new ObjectInputStream(bis); loDeserialized = (ListObject) in.readObject(); - } - catch(Exception e){ + } catch (Exception e) { System.out.println("Error while serializing and deserializing to bytes: " + e); - assert(false); + assert (false); } MatrixObject mo1Deserialized = (MatrixObject) loDeserialized.getData(0); @@ -122,6 +125,17 @@ public void serializeListObjectProgramConverter() { Assert.assertEquals(io.getLongValue(), actualIO.getLongValue()); } + @Test + public void serializeFrameObjectProgramConverter() { + FrameObject fo = new FrameObject("test"); + fo.setMetaData(new MetaDataFormat(new MatrixCharacteristics(10, 2), Types.FileFormat.BINARY)); + fo.setColumnNames(new String[]{"A", "B"}); + String serialized = ProgramConverter.serializeDataObject("X", fo); + Object[] parsed = ProgramConverter.parseDataObject(serialized); + FrameObject restored =(FrameObject) parsed[1]; + Assert.assertArrayEquals(new String[]{"A", "B"}, restored.getColumnNames()); + } + public static MatrixObject generateDummyMatrix(int size) { double[] dl = new double[size]; for (int i = 0; i < size; i++) { diff --git a/src/test/java/org/apache/sysds/test/functions/frame/FrameColNamesPropagationTest.java b/src/test/java/org/apache/sysds/test/functions/frame/FrameColNamesPropagationTest.java new file mode 100644 index 00000000000..393e74fb51f --- /dev/null +++ b/src/test/java/org/apache/sysds/test/functions/frame/FrameColNamesPropagationTest.java @@ -0,0 +1,327 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.apache.sysds.test.functions.frame; + +import java.io.IOException; +import java.util.Arrays; +import java.util.Collection; +import java.util.Collections; + +import org.apache.sysds.api.DMLScript; +import org.apache.sysds.common.Types; +import org.apache.sysds.common.Types.ExecType; +import org.apache.sysds.common.Types.FileFormat; +import org.apache.sysds.runtime.frame.data.FrameBlock; +import org.apache.sysds.runtime.io.FileFormatPropertiesCSV; +import org.apache.sysds.runtime.io.FrameWriter; +import org.apache.sysds.runtime.io.FrameWriterFactory; +import org.apache.sysds.test.AutomatedTestBase; +import org.apache.sysds.test.TestConfiguration; +import org.apache.sysds.test.TestUtils; +import org.junit.Assert; +import org.junit.Test; +import org.junit.runner.RunWith; +import org.junit.runners.Parameterized; + + +@RunWith(value = Parameterized.class) +@net.jcip.annotations.NotThreadSafe +public class FrameColNamesPropagationTest extends AutomatedTestBase { + private final static String TEST_NAME_CBIND = "ColNameCbindPropagation"; + private final static String TEST_NAME_RBIND = "ColNameRbindPropagation"; + private final static String TEST_NAME_SLICE = "ColNameSlicePropagation"; + private final static String TEST_NAME_LEFT_INDEXING = "ColNameLeftIndexingPropagation"; + private final static String TEST_DIR = "functions/frame/"; + private static final String TEST_CLASS_DIR = TEST_DIR + FrameColNamesPropagationTest.class.getSimpleName() + "/"; + + @Parameterized.Parameter + public int _matrixDim; + + @Parameterized.Parameters + public static Collection data() { + return Arrays.asList(new Object[][]{ + {10}, + {100}, + {1000}, + {2500}, + }); + } + + @Override + public void setUp() { + addTestConfiguration(TEST_NAME_CBIND, new TestConfiguration(TEST_CLASS_DIR, TEST_NAME_CBIND, new String[]{"B"})); + addTestConfiguration(TEST_NAME_RBIND, new TestConfiguration(TEST_CLASS_DIR, TEST_NAME_RBIND, new String[]{"B"})); + addTestConfiguration(TEST_NAME_SLICE, new TestConfiguration(TEST_CLASS_DIR, TEST_NAME_SLICE, new String[]{"B"})); + addTestConfiguration(TEST_NAME_LEFT_INDEXING, new TestConfiguration(TEST_CLASS_DIR, TEST_NAME_LEFT_INDEXING, new String[]{"B"})); + } + + private void writeInputFrame(String name, int rows, int cols, String prefix, long seed) throws IOException { + String[] columnNames = genColnames(cols, prefix); + + Types.ValueType[] schema = Collections.nCopies(cols, Types.ValueType.FP64).toArray(new Types.ValueType[0]); + + FrameBlock frame = new FrameBlock(schema); + frame.setColumnNames(columnNames); + + double[][] data = getRandomMatrix(rows, cols, -10, 10, 0.7, seed); + + TestUtils.initFrameData(frame, data, schema, rows); + + FrameWriter writer = FrameWriterFactory.createFrameWriter(FileFormat.CSV, + new FileFormatPropertiesCSV(true, ",", false)); + + writer.writeFrameToHDFS(frame, input(name), rows, cols); + } + + @Test + public void testPropagationCbindCP() { + runPropagationCbindTest(_matrixDim, ExecType.CP); + } + + @Test + public void testPropagationRbindCP() { + runPropagationRbindTest(_matrixDim, ExecType.CP); + } + + @Test + public void testPropagationSliceCP() { + runPropagationSliceTest(_matrixDim, ExecType.CP); + } + + @Test + public void testPropagationLeftIndexingCP() { + runPropagationLeftIndexingTest(_matrixDim, ExecType.CP); + } + + @Test + public void testPropagationCbindSpark() { + runPropagationCbindTest(_matrixDim, ExecType.SPARK); + } + + @Test + public void testPropagationRbindSpark() { + runPropagationRbindTest(_matrixDim, ExecType.SPARK); + } + + @Test + public void testPropagationSliceSpark() { + runPropagationSliceTest(_matrixDim, ExecType.SPARK); + } + + @Test + public void testPropagationLeftIndexingSpark() { + runPropagationLeftIndexingTest(_matrixDim, ExecType.SPARK); + } + + + private String[] genColnames(int n, String prefix) { + String[] colName = new String[n]; + for (int i = 0; i < n; i++) { + colName[i] = prefix + i; + } + return colName; + } + + private void runPropagationCbindTest(Integer matrixDim, ExecType et) { + Types.ExecMode platformOld = setExecMode(et); + boolean sparkConfigOld = DMLScript.USE_LOCAL_SPARK_CONFIG; + setOutputBuffering(true); + try { + + // generate an array of column names depending on the dimension of the frame block + String[] colNames1 = genColnames(matrixDim, "A"); + String[] colNames2 = genColnames(matrixDim, "B"); + + getAndLoadTestConfiguration(TEST_NAME_CBIND); + String HOME = SCRIPT_DIR + TEST_DIR; + fullDMLScriptName = HOME + TEST_NAME_CBIND + ".dml"; + + + programArgs = new String[]{"-args", + input("X1_" + matrixDim), String.valueOf(matrixDim), + String.valueOf(matrixDim), + input("X2_" + matrixDim), + Integer.toString(matrixDim), + output("B")}; + + writeInputFrame("X1_" + matrixDim, matrixDim, matrixDim, "A", 14123); + + writeInputFrame("X2_" + matrixDim, matrixDim, matrixDim, "B", 14124); + + runTest(true, false, null, -1); + + FrameBlock out = readDMLFrameFromHDFS("B", FileFormat.BINARY); + + // create array of expected column names + String[] expected = new String[colNames1.length + colNames2.length]; + System.arraycopy(colNames1, 0, expected, 0, colNames1.length); + System.arraycopy(colNames2, 0, expected, colNames1.length, colNames2.length); + + // compare column names after operation with expected column names + for (int i = 0; i < expected.length; i++) { + Assert.assertEquals( + "Wrong colName at pos:" + i, + expected[i], + out.get(0, i).toString() + ); + } + + } catch (Exception ex) { + throw new RuntimeException(ex); + } finally { + rtplatform = platformOld; + DMLScript.USE_LOCAL_SPARK_CONFIG = sparkConfigOld; + } + } + + private void runPropagationRbindTest(Integer matrixDim, ExecType et) { + Types.ExecMode platformOld = setExecMode(et); + boolean sparkConfigOld = DMLScript.USE_LOCAL_SPARK_CONFIG; + setOutputBuffering(true); + try { + + // generate an array of column names depending on the dimension of the frame block + String[] colNames1 = genColnames(matrixDim, "A"); + + getAndLoadTestConfiguration(TEST_NAME_RBIND); + String HOME = SCRIPT_DIR + TEST_DIR; + fullDMLScriptName = HOME + TEST_NAME_RBIND + ".dml"; + + + programArgs = new String[]{"-args", + input("X1_" + matrixDim), String.valueOf(matrixDim), + String.valueOf(matrixDim), + input("X2_" + matrixDim), + Integer.toString(matrixDim), + output("B")}; + + writeInputFrame("X1_" + matrixDim, matrixDim, matrixDim, "A", 14123); + + writeInputFrame("X2_" + matrixDim, matrixDim, matrixDim, "B", 14124); + + runTest(true, false, null, -1); + + FrameBlock out = readDMLFrameFromHDFS("B", FileFormat.BINARY); + + // expected are the column names from the first frame block + for (int i = 0; i < colNames1.length; i++) { + Assert.assertEquals( + "Wrong colName at pos:" + i, + colNames1[i], + out.get(0, i).toString() + ); + } + + } catch (Exception ex) { + throw new RuntimeException(ex); + } finally { + rtplatform = platformOld; + DMLScript.USE_LOCAL_SPARK_CONFIG = sparkConfigOld; + } + } + + private void runPropagationSliceTest(Integer matrixDim, ExecType et) { + Types.ExecMode platformOld = setExecMode(et); + boolean sparkConfigOld = DMLScript.USE_LOCAL_SPARK_CONFIG; + setOutputBuffering(true); + try { + + // generate an array of column names depending on the dimension of the frame block + String[] colNames = genColnames(matrixDim, "A"); + + getAndLoadTestConfiguration(TEST_NAME_SLICE); + String HOME = SCRIPT_DIR + TEST_DIR; + fullDMLScriptName = HOME + TEST_NAME_SLICE + ".dml"; + + + programArgs = new String[]{"-args", + input("X_" + matrixDim), String.valueOf(matrixDim), + String.valueOf(matrixDim), + output("B")}; + + writeInputFrame("X1", matrixDim, matrixDim, "A", 14123); + + runTest(true, false, null, -1); + + FrameBlock out = readDMLFrameFromHDFS("B", FileFormat.BINARY); + + String[] expected = Arrays.copyOfRange(colNames, 1, colNames.length - 1); + + // expected are the sliced column names + for (int i = 0; i < expected.length; i++) { + Assert.assertEquals( + "Wrong colName at pos:" + i, + expected[i], + out.get(0, i).toString() + ); + } + + } catch (Exception ex) { + throw new RuntimeException(ex); + } finally { + rtplatform = platformOld; + DMLScript.USE_LOCAL_SPARK_CONFIG = sparkConfigOld; + } + } + + private void runPropagationLeftIndexingTest(int matrixDim, ExecType et) { + Types.ExecMode platformOld = setExecMode(et); + boolean sparkConfigOld = DMLScript.USE_LOCAL_SPARK_CONFIG; + setOutputBuffering(true); + try { + + // generate an array of column names depending on the dimension of the frame block + String[] colNames = genColnames(matrixDim, "A"); + + getAndLoadTestConfiguration(TEST_NAME_LEFT_INDEXING); + String HOME = SCRIPT_DIR + TEST_DIR; + fullDMLScriptName = HOME + TEST_NAME_LEFT_INDEXING + ".dml"; + + + programArgs = new String[]{"-args", + input("X_" + matrixDim), String.valueOf(matrixDim), + String.valueOf(matrixDim), + output("B")}; + + writeInputFrame("X_" + matrixDim, matrixDim, matrixDim, "A", 14123); + + runTest(true, false, null, -1); + + FrameBlock out = readDMLFrameFromHDFS("B", FileFormat.BINARY); + + String[] expected = colNames; + + for (int i = 0; i < expected.length; i++) { + Assert.assertEquals( + "Wrong colName at pos:" + i, + expected[i], + out.get(0, i).toString() + ); + } + + } catch (Exception ex) { + throw new RuntimeException(ex); + } finally { + rtplatform = platformOld; + DMLScript.USE_LOCAL_SPARK_CONFIG = sparkConfigOld; + } + } +} + diff --git a/src/test/java/org/apache/sysds/test/functions/frame/FrameColumnNamesTest.java b/src/test/java/org/apache/sysds/test/functions/frame/FrameColumnNamesTest.java index d1ee4215e1a..e72ba519d81 100644 --- a/src/test/java/org/apache/sysds/test/functions/frame/FrameColumnNamesTest.java +++ b/src/test/java/org/apache/sysds/test/functions/frame/FrameColumnNamesTest.java @@ -44,6 +44,8 @@ @net.jcip.annotations.NotThreadSafe public class FrameColumnNamesTest extends AutomatedTestBase { private final static String TEST_NAME = "ColumnNames"; + private final static String TEST_NAME_GET = "GetNames"; + private final static String TEST_NAME_SET = "SetNames"; private final static String TEST_DIR = "functions/frame/"; private static final String TEST_CLASS_DIR = TEST_DIR + FrameColumnNamesTest.class.getSimpleName() + "/"; @@ -60,6 +62,9 @@ public static Collection data() { @Override public void setUp() { addTestConfiguration(TEST_NAME, new TestConfiguration(TEST_CLASS_DIR, TEST_NAME, new String[] {"B"})); + addTestConfiguration(TEST_NAME_GET, new TestConfiguration(TEST_CLASS_DIR, TEST_NAME_GET, new String[] {"B"})); + addTestConfiguration(TEST_NAME_SET, new TestConfiguration(TEST_CLASS_DIR, TEST_NAME_SET, new String[] {"B"})); + } @Test @@ -72,6 +77,134 @@ public void testDetectSchemaDoubleSpark() { runGetColNamesTest(_columnNames, ExecType.SPARK); } + @Test + public void testGetNamesCP() { + runGetNamesTest(_columnNames, ExecType.CP); + } + + @Test + public void testGetNamesSpark() { + runGetNamesTest(_columnNames, ExecType.SPARK); + } + + @Test + public void testSetNamesCP() { + runSetNamesTest(_columnNames, ExecType.CP); + } + + @Test + public void testSetNamesSpark() { + runSetNamesTest(_columnNames, ExecType.SPARK); + } + + private void runGetNamesTest(String[] columnNames, ExecType et) { + Types.ExecMode platformOld = setExecMode(et); + boolean sparkConfigOld = DMLScript.USE_LOCAL_SPARK_CONFIG; + + if(et == ExecType.SPARK) + DMLScript.USE_LOCAL_SPARK_CONFIG = true; + setOutputBuffering(true); + try { + getAndLoadTestConfiguration(TEST_NAME_GET); + String HOME = SCRIPT_DIR + TEST_DIR; + fullDMLScriptName = HOME + TEST_NAME_GET + ".dml"; + programArgs = new String[] {"-args", input("A"), String.valueOf(_rows), + Integer.toString(columnNames.length), output("B")}; + + Types.ValueType[] schema = Collections.nCopies( + columnNames.length, Types.ValueType.FP64).toArray(new Types.ValueType[0]); + FrameBlock frame1 = new FrameBlock(schema); + frame1.setColumnNames(columnNames); + FrameWriter writer = FrameWriterFactory.createFrameWriter(FileFormat.CSV, + new FileFormatPropertiesCSV(true, ",", false)); + + double[][] A = getRandomMatrix(_rows, schema.length, Double.MIN_VALUE, Double.MAX_VALUE, 0.7, 14123); + TestUtils.initFrameData(frame1, A, schema, _rows); + writer.writeFrameToHDFS(frame1, input("A"), _rows, schema.length); + + runTest(true, false, null, -1); + + FrameBlock resultFrame = + readDMLFrameFromHDFS("B", FileFormat.BINARY); + + Assert.assertEquals( + "Unexpected number of result rows.", + 1, + resultFrame.getNumRows()); + + Assert.assertEquals( + "Unexpected number of result columns.", + columnNames.length, + resultFrame.getNumColumns()); + + // verify output schema + for(int i = 0; i < schema.length; i++) { + Assert + .assertEquals("Wrong result: " + columnNames[i] + ".", columnNames[i], resultFrame.get(0, i).toString()); + } + } + catch(Exception ex) { + throw new RuntimeException(ex); + } + finally { + rtplatform = platformOld; + DMLScript.USE_LOCAL_SPARK_CONFIG = sparkConfigOld; + } + } + + private void runSetNamesTest(String[] columnNames, ExecType et) { + Types.ExecMode platformOld = setExecMode(et); + boolean sparkConfigOld = DMLScript.USE_LOCAL_SPARK_CONFIG; + + if(et == ExecType.SPARK) + DMLScript.USE_LOCAL_SPARK_CONFIG = true; + setOutputBuffering(true); + try { + getAndLoadTestConfiguration(TEST_NAME_SET); + String HOME = SCRIPT_DIR + TEST_DIR; + fullDMLScriptName = HOME + TEST_NAME_SET + ".dml"; + programArgs = new String[] {"-args",input("X"),String.valueOf(_rows),Integer.toString(columnNames.length), + input("N"),output("B") + }; + + Types.ValueType[] schema = Collections.nCopies( + columnNames.length, Types.ValueType.FP64).toArray(new Types.ValueType[0]); + + FrameBlock frame1 = new FrameBlock(schema); + FrameWriter writer = FrameWriterFactory.createFrameWriter(FileFormat.CSV, + new FileFormatPropertiesCSV(true, ",", false)); + + double[][] A = getRandomMatrix(_rows, schema.length, Double.MIN_VALUE, Double.MAX_VALUE, 0.7, 14123); + TestUtils.initFrameData(frame1, A, schema, _rows); + writer.writeFrameToHDFS(frame1, input("X"), _rows, schema.length); + + Types.ValueType[] nameSchema = Collections.nCopies( + columnNames.length, Types.ValueType.STRING).toArray(new Types.ValueType[0]); + + FrameBlock names = new FrameBlock(nameSchema); + names.ensureAllocatedColumns(1); + for(int i = 0; i < columnNames.length; i++) + names.set(0, i, columnNames[i]); + FrameWriter nameWriter = FrameWriterFactory.createFrameWriter(FileFormat.CSV, + new FileFormatPropertiesCSV(false, ",", false)); + nameWriter.writeFrameToHDFS(names, input("N"), 1, columnNames.length); + + runTest(true, false, null, -1); + + FrameBlock frame2 = readDMLFrameFromHDFS("B", FileFormat.BINARY); + for(int i = 0; i < columnNames.length; i++) + Assert.assertEquals("Wrong result: " + columnNames[i] + ".", columnNames[i], frame2.get(0, i).toString()); + } + catch(Exception ex) { + throw new RuntimeException(ex); + } + finally { + rtplatform = platformOld; + DMLScript.USE_LOCAL_SPARK_CONFIG = sparkConfigOld; + } + } + + private void runGetColNamesTest(String[] columnNames, ExecType et) { Types.ExecMode platformOld = setExecMode(et); boolean sparkConfigOld = DMLScript.USE_LOCAL_SPARK_CONFIG; diff --git a/src/test/scripts/functions/frame/ColNameCbindPropagation.dml b/src/test/scripts/functions/frame/ColNameCbindPropagation.dml new file mode 100644 index 00000000000..e46a4a7dbe8 --- /dev/null +++ b/src/test/scripts/functions/frame/ColNameCbindPropagation.dml @@ -0,0 +1,26 @@ +#------------------------------------------------------------- +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +# +#------------------------------------------------------------- + +X1 = read($1, rows=$2, cols=$3, data_type="frame", format="csv", header=TRUE); +X2 = read($4, rows=$2, cols=$5, data_type="frame", format="csv", header=TRUE); +Y = cbind(X1, X2); +B = getNames(Y); +write(B, $6, format="binary"); \ No newline at end of file diff --git a/src/test/scripts/functions/frame/ColNameLeftIndexingPropagation.dml b/src/test/scripts/functions/frame/ColNameLeftIndexingPropagation.dml new file mode 100644 index 00000000000..0c539d3a294 --- /dev/null +++ b/src/test/scripts/functions/frame/ColNameLeftIndexingPropagation.dml @@ -0,0 +1,26 @@ +#------------------------------------------------------------- +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +# +#------------------------------------------------------------- + +X = read($1, rows=$2, cols=$3, data_type="frame", format="csv", header=TRUE); +Y = X; +Y[1,1] = "replacement"; +B = getNames(Y); +write(B, $4, format="binary"); diff --git a/src/test/scripts/functions/frame/ColNameRbindPropagation.dml b/src/test/scripts/functions/frame/ColNameRbindPropagation.dml new file mode 100644 index 00000000000..e11892f3645 --- /dev/null +++ b/src/test/scripts/functions/frame/ColNameRbindPropagation.dml @@ -0,0 +1,26 @@ +#------------------------------------------------------------- +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +# +#------------------------------------------------------------- + +X1 = read($1, rows=$2, cols=$3, data_type="frame", format="csv", header=TRUE); +X2 = read($4, rows=$2, cols=$5, data_type="frame", format="csv", header=TRUE); +Y = rbind(X1, X2); +B = getNames(Y); +write(B, $6, format="binary"); \ No newline at end of file diff --git a/src/test/scripts/functions/frame/ColNameSlicePropagation.dml b/src/test/scripts/functions/frame/ColNameSlicePropagation.dml new file mode 100644 index 00000000000..647e4f172d5 --- /dev/null +++ b/src/test/scripts/functions/frame/ColNameSlicePropagation.dml @@ -0,0 +1,25 @@ +#------------------------------------------------------------- +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +# +#------------------------------------------------------------- + +X = read($1, rows=$2, cols=$3, data_type="frame", format="csv", header=TRUE); +Y = X[,2:($3-1)]; +B = getNames(Y); +write(B, $4, format="binary"); \ No newline at end of file diff --git a/src/test/scripts/functions/frame/GetNames.dml b/src/test/scripts/functions/frame/GetNames.dml new file mode 100644 index 00000000000..70b8f22d8d9 --- /dev/null +++ b/src/test/scripts/functions/frame/GetNames.dml @@ -0,0 +1,24 @@ +#------------------------------------------------------------- +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +# +#------------------------------------------------------------- + +X = read($1, rows=$2, cols=$3, data_type="frame", format="csv", header=TRUE); +R = getNames(X); +write(R, $4, format="binary"); \ No newline at end of file diff --git a/src/test/scripts/functions/frame/SetNames.dml b/src/test/scripts/functions/frame/SetNames.dml new file mode 100644 index 00000000000..157a415babc --- /dev/null +++ b/src/test/scripts/functions/frame/SetNames.dml @@ -0,0 +1,28 @@ +#------------------------------------------------------------- +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +# +#------------------------------------------------------------- + +X = read($1, rows=$2, cols=$3, data_type="frame", format="csv", header=TRUE); +N = read($4, rows=1, cols=$3, data_type="frame", format="csv", header=FALSE); + +X2 = setNames(X, N) +B = getNames(X2) + +write(B, $5, format="binary"); \ No newline at end of file