ds.granules.operation

Class MapReduceBase

java.lang.Object

ds.granules.operation.OperationBase

ds.granules.operation.MapReduceBase

All Implemented Interfaces:

FormatMarshaller, Map, MapReduce, Operation, OperationConfiguration, Reduce

Direct Known Subclasses:

GIComputation, UpperCaseMap, WordCountMap, WordCountReduce

public abstract class MapReduceBase
extends OperationBase
implements Map, Reduce

This is the base class which will serve as the base for implementations of the Map and Reduce interface.

Author:

Shrideep Pallickara

Nested Class Summary

Nested classes/interfaces inherited from interface ds.granules.operation.Operation

Operation.OperationType

Field Summary

Fields

Modifier and Type	Field and Description
static java.lang.String	MODULE

Constructor Summary

Constructors

Constructor and Description
MapReduceBase()

Method Summary

Modifier and Type	Method and Description
int	addMap(Map map) Adds a Map that will be reduced.
int	addReduce(Reduce reduce) Adds a Reduce for this particular Map.
Results	createResults(boolean completeResult, boolean processingComplete) Create the Results type to be used for reporting results produced as a result of the processing.
java.util.Iterator<java.lang.String>	getAwaitingOutputsFrom() Retrieves the set of Map identifiers from which an Output has not been received so far.
DatasetCollection	getDatasetCollection() Retrieve the dataset that is associated with this particular operation.
java.util.Enumeration<java.lang.String>	getMapIdentifiers() Retrieves the set of Map identifiers
int	getNumberOfMaps() Retrieves the number of registered Map instances
int	getNumberOfReducers() Retrieves the number of registered Reduce instances
java.util.Enumeration<java.lang.String>	getReduceIdentifiers() Retrieves the set of Reduce identifiers
boolean	isAwaitingOutputs() Check to see is the Reducer is missing outputs from any of the registered Maps.
void	marshall(FormatWriter mFormat)
int	removeMap(java.lang.String mapIdentifier) Removes a Map from the list of Maps that are registered to this reducer.
int	removeReduce(java.lang.String reduceIdentifier) Removes a given reducer from the list of Reducers for a given Map function.
void	resetOutputTracker() This method resets the output tracker associated with the reducer.
void	setReceivedOutputFrom(java.lang.String mapIdentifier) Mark the receipt of Output from a given Map instance
void	unmarshall(FormatReader mFormat) Unmarshall the operation from a set of marshalled bytes.
void	writeResults(int synopsisType, java.lang.String streamSynopsis, Results results) Deprecated.
void	writeResults(Results results) Writes the specified results to the default output stream.
void	writeResults(java.lang.String streamSynopsis, Results results) Deprecated.
void	writeResults(Topic topic, Results results) Writes the specified results to a specific output stream.

Methods inherited from class ds.granules.operation.OperationBase

dispose, execute, getDomain, getExecutionProfile, getInstanceIdentifier, getOperationIdentifier, getProcessingDirectives, getVersionInformation, hasDatasetCollection, hasExecutionProfile, hasProcessingDirectives, initialize, isInitialized, markInitializationAsComplete, needsInitialization, setAsExecuteFixedNumberOfTimes, setAsExecuteOnce, setAsExecutePeriodically, setAsExecuteWhenDataAvailable, setAsStayAlive, setDatasetCollection, setDomain, setInitializationComplete, setInitializationNeeded, setInstanceIdentifier, setOperationIdentifier, setProcessingDirectives, setTeminationConditionReached, setVersionInformation, terminationConditionReached, toString

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, wait, wait, wait

Methods inherited from interface ds.granules.operation.Operation

dispose, execute, getDomain, getExecutionProfile, getInstanceIdentifier, getOperationIdentifier, getProcessingDirectives, getVersionInformation, hasDatasetCollection, hasExecutionProfile, hasProcessingDirectives, initialize, isInitialized, needsInitialization, terminationConditionReached, toString

Field Detail

MODULE

public static final java.lang.String MODULE

See Also:

Constant Field Values

Constructor Detail

MapReduceBase

public MapReduceBase()

Method Detail

unmarshall

public void unmarshall(FormatReader mFormat)
                throws FormatException

Description copied from interface: Operation

Unmarshall the operation from a set of marshalled bytes.

Specified by:

unmarshall in interface Operation

Overrides:

unmarshall in class OperationBase

Parameters:

mFormat - The byte[] representation of the operation.

Throws:

FormatException

marshall

public void marshall(FormatWriter mFormat)
              throws FormatException

Specified by:

marshall in interface FormatMarshaller

Overrides:

marshall in class OperationBase

Parameters:

mFormat - - The format writer to write to

Throws:

FormatException - If there was a problem while writing

getDatasetCollection

public DatasetCollection getDatasetCollection()

Description copied from interface: Operation

Retrieve the dataset that is associated with this particular operation.

Specified by:

getDatasetCollection in interface Operation

Overrides:

getDatasetCollection in class OperationBase

Returns:

The underlying dataset collection.

addReduce

public int addReduce(Reduce reduce)
              throws MapReduceException

Description copied from interface: Map

Adds a Reduce for this particular Map. Typically, a given Map instance will have only one Reduce. There are special instances when multiple Reducers would be needed for a given Map function.

Specified by:

addReduce in interface Map

Parameters:

reduce - The Reducer that will be added to this Map

Returns:

The number of reducers for this Map.

Throws:

MapReduceException - If there are problems in adding this Reduce

removeReduce

public int removeReduce(java.lang.String reduceIdentifier)
                 throws MapReduceException

Description copied from interface: Map

Removes a given reducer from the list of Reducers for a given Map function.

Specified by:

removeReduce in interface Map

Parameters:

reduceIdentifier - The identifier of the Reduce to be removed

Returns:

The number of reducers for this Map, after the successful removal

Throws:

MapReduceException - If there are problems during removal.

getNumberOfReducers

public int getNumberOfReducers()

Description copied from interface: Map

Retrieves the number of registered Reduce instances

Specified by:

getNumberOfReducers in interface Map

Returns:

The number of reducers for this map.

getReduceIdentifiers

public java.util.Enumeration<java.lang.String> getReduceIdentifiers()

Description copied from interface: Map

Retrieves the set of Reduce identifiers

Specified by:

getReduceIdentifiers in interface Map

Returns:

An enumeration of the list of registered identifiers

addMap

public int addMap(Map map)
           throws MapReduceException

Description copied from interface: Reduce

Adds a Map that will be reduced.

Specified by:

addMap in interface Reduce

Parameters:

map - Add a Map instance for this reducer

Returns:

The number of maps that are registered with this reducer

Throws:

MapReduceException

removeMap

public int removeMap(java.lang.String mapIdentifier)
              throws MapReduceException

Description copied from interface: Reduce

Removes a Map from the list of Maps that are registered to this reducer.

Specified by:

removeMap in interface Reduce

Parameters:

mapIdentifier - The identifier of the Map to removed

Returns:

The number of maps that are registered with this reducer

Throws:

MapReduceException - If there are problems removing the Map.

getMapIdentifiers

public java.util.Enumeration<java.lang.String> getMapIdentifiers()

Description copied from interface: Reduce

Retrieves the set of Map identifiers

Specified by:

getMapIdentifiers in interface Reduce

Returns:

An enumeration of the list of registered identifiers

getNumberOfMaps

public int getNumberOfMaps()

Description copied from interface: Reduce

Retrieves the number of registered Map instances

Specified by:

getNumberOfMaps in interface Reduce

Returns:

The number of Maps

isAwaitingOutputs

public boolean isAwaitingOutputs()

Description copied from interface: Reduce

Check to see is the Reducer is missing outputs from any of the registered Maps.

Specified by:

isAwaitingOutputs in interface Reduce

Returns:

true is the Reducer is still awaiting outputs; false otherwise.

setReceivedOutputFrom

public void setReceivedOutputFrom(java.lang.String mapIdentifier)

Description copied from interface: Reduce

Mark the receipt of Output from a given Map instance

Specified by:

setReceivedOutputFrom in interface Reduce

Parameters:

mapIdentifier - The ID of the Map from which an output has been received.

getAwaitingOutputsFrom

public java.util.Iterator<java.lang.String> getAwaitingOutputsFrom()

Description copied from interface: Reduce

Retrieves the set of Map identifiers from which an Output has not been received so far.

Specified by:

getAwaitingOutputsFrom in interface Reduce

Returns:

An iterator of the pending identifiers

resetOutputTracker

public void resetOutputTracker()

Description copied from interface: Reduce

This method resets the output tracker associated with the reducer. This is useful when the stages are part of an iterative computation.

Specified by:

resetOutputTracker in interface Reduce

createResults

public Results createResults(boolean completeResult,
                             boolean processingComplete)

Description copied from interface: MapReduce

Create the Results type to be used for reporting results produced as a result of the processing.

Specified by:

createResults in interface MapReduce

Returns:

The created Results.

writeResults

public void writeResults(Results results)
                  throws MapReduceException

Description copied from interface: MapReduce

Writes the specified results to the default output stream.

Specified by:

writeResults in interface MapReduce

Parameters:

results - The results to be written.

Throws:

MapReduceException - If there are problems in marshalling or writing these results.

writeResults

public void writeResults(Topic topic,
                         Results results)
                  throws MapReduceException

Description copied from interface: MapReduce

Writes the specified results to a specific output stream.

Specified by:

writeResults in interface MapReduce

Parameters:

topic - The topic to write results to

results - The results that needs to be written

Throws:

MapReduceException - If there are problems in marshalling or writing these results.

writeResults

@Deprecated
public void writeResults(java.lang.String streamSynopsis,
                                     Results results)
                              throws MapReduceException

Deprecated.

Description copied from interface: MapReduce

Writes the specified results to a specific output stream.

Specified by:

writeResults in interface MapReduce

Parameters:

streamSynopsis - The synopsis for the stream

results - The results that needs to be written

Throws:

MapReduceException - If there are problems in marshalling or writing these results.

writeResults

@Deprecated
public void writeResults(int synopsisType,
                                     java.lang.String streamSynopsis,
                                     Results results)
                              throws MapReduceException

Deprecated.

Description copied from interface: MapReduce

Writes the specified results to a specific output stream.

Specified by:

writeResults in interface MapReduce

Parameters:

synopsisType - The synopsis type for the stream in question

streamSynopsis - The synopsis for the stream

results - The results that needs to be written

Throws:

MapReduceException - If there are problems in marshalling or writing these results.