Skip to content

Instantly share code, notes, and snippets.

View jkschneider's full-sized avatar

Jonathan Schnéider jkschneider

View GitHub Profile
@kaniska
kaniska / SparkNLP_SparkML_Similarity_Test.scala
Created September 6, 2020 03:02
Semantic Similarity using Spark NLP and Spark ML
// Databricks notebook source
import com.johnsnowlabs.nlp.annotators.Tokenizer
import com.johnsnowlabs.nlp.annotators.sbd.pragmatic.SentenceDetector
import com.johnsnowlabs.nlp.embeddings.{BertEmbeddings, SentenceEmbeddings, WordEmbeddingsModel}
import com.johnsnowlabs.nlp.{DocumentAssembler, EmbeddingsFinisher, RecursivePipeline}
import org.apache.spark.ml.PipelineModel
import org.apache.spark.ml.feature.{BucketedRandomProjectionLSH, BucketedRandomProjectionLSHModel, LSH, Normalizer, SQLTransformer}
import org.apache.spark.ml.feature.{MinHashLSH, MinHashLSHModel}
import org.apache.spark.sql.SparkSession
import org.apache.spark.sql.functions._
import lombok.AccessLevel;
import lombok.EqualsAndHashCode;
import lombok.Getter;
import lombok.RequiredArgsConstructor;
import lombok.Value;
import lombok.extern.slf4j.Slf4j;
import org.reactivestreams.Publisher;
import org.springframework.core.io.buffer.DataBuffer;
import org.springframework.core.io.buffer.DefaultDataBufferFactory;
import reactor.core.publisher.Flux;
@mcxiaoke
mcxiaoke / git-show-big-files.sh
Created January 5, 2018 07:22 — forked from debreczeni/git-show-big-files.sh
Find large files in git repository
#!/bin/bash
#set -x
# Shows you the largest objects in your repo's pack file.
# Written for osx.
#
# @see http://stubbisms.wordpress.com/2009/07/10/git-script-to-show-largest-pack-objects-and-trim-your-waist-line/
# @author Antony Stubbs
@DarrenForsythe
DarrenForsythe / ClassLoaderLeakExample.java
Created September 27, 2017 19:02 — forked from dpryden/ClassLoaderLeakExample.java
Example of a ClassLoader leak in Java
import java.io.IOException;
import java.net.URLClassLoader;
import java.nio.file.Files;
import java.nio.file.Paths;
import java.nio.file.Path;
/**
* Example demonstrating a ClassLoader leak.
*
* <p>To see it in action, copy this file to a temp directory somewhere,