Category : flink

Order This article mainly studies flink’s log.file configuration. log4j.properties flink-release-1.6.2/flink-dist/src/main/flink-bin/conf/log4j.properties # This affects logging for both user code and Flink log4j.rootLogger=INFO, file # Uncomment this if you want to _only_ change Flink’s logging #log4j.logger.org.apache.flink=INFO # The following lines keep the log level of common libraries/connectors on # log level INFO. The root logger does not ..

Read more

Order This article mainly studies how flink is compatible with StormTopology. Example @Test public void testStormWordCount() throws Exception { //NOTE 1 build Topology the Storm way final TopologyBuilder builder = new TopologyBuilder(); builder.setSpout(“spout”, new RandomWordSpout(), 1); builder.setBolt(“count”, new WordCountBolt(), 5) .fieldsGrouping(“spout”, new Fields(“word”)); builder.setBolt(“print”, new PrintBolt(), 1) .shuffleGrouping(“count”); //NOTE 2 convert StormTopology to FlinkTopology FlinkTopology ..

Read more

Order This article mainly studies flink’s SpoutWrapper SpoutWrapper flink-storm_2.11-1.6.2-sources.jar! /org/apache/flink/storm/wrappers/SpoutWrapper.java /** * A {@link SpoutWrapper} wraps an {@link IRichSpout} in order to execute it within a Flink Streaming program. It * takes the spout’s output tuples and transforms them into Flink tuples of type {@code OUT} (see * {@link SpoutCollector} for supported types).<br> * <br> ..

Read more

Order This article mainly studies flink’s BoltWrapper. BoltWrapper flink-storm_2.11-1.6.2-sources.jar! /org/apache/flink/storm/wrappers/BoltWrapper.java /** * A {@link BoltWrapper} wraps an {@link IRichBolt} in order to execute the Storm bolt within a Flink Streaming program. * It takes the Flink input tuples of type {@code IN} and transforms them into {@link StormTuple}s that the bolt can * process. Furthermore, ..

Read more

Order This article mainly studies flink’s CsvReader Example final ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment(); DataSet<RecordDto> csvInput = env.readCsvFile(csvFilePath) .pojoType(RecordDto.class, “playerName”, “country”, “year”, “game”, “gold”, “silver”, “bronze”, “total”); csvInput.map(new MapFunction<RecordDto, RecordDto>() { @Override public RecordDto map(RecordDto value) throws Exception { LOGGER.info(“execute map:{}”,value); TimeUnit.SECONDS.sleep(5); return value; } }).print(); ExecutionEnvironment.readCsvFile flink-java-1.6.2-sources.jar! /org/apache/flink/api/java/ExecutionEnvironment.java /** * Creates a CSV reader to ..

Read more

Order This article mainly studies flink’s SourceFunction. Example // set up the execution environment final StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment(); DataStreamSource<String> dataStreamSource = env.addSource(new RandomWordSource()); dataStreamSource.map(new UpperCaseMapFunc()).print(); env.execute(“sourceFunctionDemo”); A custom SourceFunction is added here through the addSource method. SourceFunction flink-streaming-java_2.11-1.6.2-sources.jar! /org/apache/flink/streaming/api/functions/source/SourceFunction.java /** * Base interface for all stream data sources in Flink. The contract of a ..

Read more

Order This article mainly studies flink’s RichParallelSourceFunction RichParallelSourceFunction /** * Base class for implementing a parallel data source. Upon execution, the runtime will * execute as many parallel instances of this function function as configured parallelism * of the source. * * <p>The data source has access to context information (such as the number of ..

Read more

Order This article mainly studies flink’s InputFormatSourceFunction Example final StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment(); IteratorInputFormat iteratorInputFormat = new IteratorInputFormat<String>(new WordIterator()); env //TypeInformation.of(new TypeHint<String>() {} .createInput(iteratorInputFormat,TypeExtractor.createTypeInfo(String.class)) .setParallelism(1) .print(); IteratorInputFormat is used here to call env’s createInput method to create SourceFunction. StreamExecutionEnvironment.createInput flink-streaming-java_2.11-1.6.2-sources.jar! /org/apache/flink/streaming/api/environment/StreamExecutionEnvironment.java @PublicEvolving public <OUT> DataStreamSource<OUT> createInput(InputFormat<OUT, ?> inputFormat, TypeInformation<OUT> typeInfo) { DataStreamSource<OUT> source; if (inputFormat ..

Read more

Order This article mainly studies flink’s ParallelIteratorInputFormat Example final ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment(); DataSet<Long> dataSet = env.generateSequence(15,106) .setParallelism(3); dataSet.print(); The generateSequence method of ExecutionEnvironment is used here to create ParallelIteratorInputFormat with NumberSequenceIterator. ParallelIteratorInputFormat flink-java-1.6.2-sources.jar! /org/apache/flink/api/java/io/ParallelIteratorInputFormat.java /** * An input format that generates data in parallel through a {@link SplittableIterator}. */ @PublicEvolving public class ParallelIteratorInputFormat<T> extends ..

Read more

Order This article mainly studies flink’s PrintSinkFunction DataStream.print flink-streaming-java_2.11-1.7.0-sources.jar! /org/apache/flink/streaming/api/datastream/DataStream.java /** * Writes a DataStream to the standard output stream (stdout). * * <p>For each element of the DataStream the result of {@link Object#toString()} is written. * * <p>NOTE: This will print to stdout on the machine where the code is executed, i.e. the Flink ..

Read more

Order This article mainly studies flink’s SocketClientSink DataStream.writeToSocket flink-streaming-java_2.11-1.7.0-sources.jar! /org/apache/flink/streaming/api/datastream/DataStream.java /** * Writes the DataStream to a socket as a byte array. The format of the * output is specified by a {@link SerializationSchema}. * * @param hostName * host of the socket * @param port * port of the socket * @param schema * ..

Read more

Order This article mainly studies flink’s TextOutputFormat DataStream.writeAsText flink-streaming-java_2.11-1.7.0-sources.jar! /org/apache/flink/streaming/api/datastream/DataStream.java /** * Writes a DataStream to the file specified by path in text format. * * <p>For every element of the DataStream the result of {@link Object#toString()} is written. * * @param path * The path pointing to the location the text file is written ..

Read more