Building Spark Applications in IDE( NeedPOM.XML dependencies)


3 Answer(s)


Hi Ramesh,

The dependencies will be given by faculty only based on spark version and other course related dependencies. I would recommend you to please wait for the class.

Thanks.

 


Hi Ramesh,

 

It is in the document Prashant shared with us; Setting up Spark Project with Scala IDE and Maven.txt

You need to update your pom.xml in the IDE tab as follows;

pom.xml for Spark projects - Copy this and add it into your project pom.xml file within the <project></project> tags
=====================================================================================================================

<pluginRepositories>
		<pluginRepository>
			<id>scala-tools.org</id>
			<name>Scala-tools Maven2 Repository</name>
			<url>http://scala-tools.org/repo-releases</url>
		</pluginRepository>
	</pluginRepositories>

	<repositories>
		<repository>
			<id>pele.farmbio.uu.se</id>
			<url>http://pele.farmbio.uu.se/artifactory/libs-snapshot</url>
		</repository>
	</repositories>

	<dependencies>
		
		<dependency>
			<groupId>org.apache.spark</groupId>
			<artifactId>spark-core_2.10</artifactId>
			<version>1.6.0</version>
		</dependency>
		
	</dependencies>

	<build>
		<plugins>
			<!-- mixed scala/java compile -->
			<plugin>
				<groupId>org.scala-tools</groupId>
				<artifactId>maven-scala-plugin</artifactId>
				<executions>
					<execution>
						<id>compile</id>
						<goals>
							<goal>compile</goal>
						</goals>
						<phase>compile</phase>
					</execution>
					<execution>
						<id>test-compile</id>
						<goals>
							<goal>testCompile</goal>
						</goals>
						<phase>test-compile</phase>
					</execution>
					<execution>
						<phase>process-resources</phase>
						<goals>
							<goal>compile</goal>
						</goals>
					</execution>
				</executions>
			</plugin>
			<plugin>
				<artifactId>maven-compiler-plugin</artifactId>
				<configuration>
					<source>1.7</source>
					<target>1.7</target>
				</configuration>
			</plugin>
			<!-- for fatjar -->
			<plugin>
				<groupId>org.apache.maven.plugins</groupId>
				<artifactId>maven-assembly-plugin</artifactId>
				<version>2.4</version>
				<configuration>
					<descriptorRefs>
						<descriptorRef>jar-with-dependencies</descriptorRef>
					</descriptorRefs>
				</configuration>
				<executions>
					<execution>
						<id>assemble-all</id>
						<phase>package</phase>
						<goals>
							<goal>single</goal>
						</goals>
					</execution>
				</executions>
			</plugin>
			<plugin>
				<groupId>org.apache.maven.plugins</groupId>
				<artifactId>maven-jar-plugin</artifactId>
				<configuration>
					<archive>
						<manifest>
							<addClasspath>true</addClasspath>
							<mainClass>fully.qualified.MainClass</mainClass>
						</manifest>
					</archive>
				</configuration>
			</plugin>
		</plugins>
		<pluginManagement>
			<plugins>
				<!--This plugin's configuration is used to store Eclipse m2e settings 
					only. It has no influence on the Maven build itself. -->
				<plugin>
					<groupId>org.eclipse.m2e</groupId>
					<artifactId>lifecycle-mapping</artifactId>
					<version>1.0.0</version>
					<configuration>
						<lifecycleMappingMetadata>
							<pluginExecutions>
								<pluginExecution>
									<pluginExecutionFilter>
										<groupId>org.scala-tools</groupId>
										<artifactId>
											maven-scala-plugin
										</artifactId>
										<versionRange>
											[2.15.2,)
										</versionRange>
										<goals>
											<goal>compile</goal>
											<goal>testCompile</goal>
										</goals>
									</pluginExecutionFilter>
									<action>
										<execute></execute>
									</action>
								</pluginExecution>
							</pluginExecutions>
						</lifecycleMappingMetadata>
					</configuration>
				</plugin>
			</plugins>
		</pluginManagement>
	</build>

Thank you Tejus...

Somehow  I missed document "Setting up Spark Project with Scala IDE and Maven.txt" .Can you please share me. My Email Id:prkr652@gmail.com