# R Interface to Apache Spark

## Documentation for package ‘sparklyr’ version 1.8.5

- DESCRIPTION file.
- Package NEWS.

## Help Pages

A C D E F G H I J L M N P R S T U misc

## -- A --

augment.ml_model_aft_survival_regression | Tidying methods for Spark ML Survival Regression
augment.ml_model_als | Tidying methods for Spark ML ALS
augment.ml_model_bisecting_kmeans | Tidying methods for Spark ML unsupervised models
augment.ml_model_decision_tree_classification | Tidying methods for Spark ML tree models
augment.ml_model_decision_tree_regression | Tidying methods for Spark ML tree models
augment.ml_model_gaussian_mixture | Tidying methods for Spark ML unsupervised models
augment.ml_model_gbt_classification | Tidying methods for Spark ML tree models
augment.ml_model_gbt_regression | Tidying methods for Spark ML tree models
augment.ml_model_generalized_linear_regression | Tidying methods for Spark ML linear models
augment.ml_model_isotonic_regression | Tidying methods for Spark ML Isotonic Regression
augment.ml_model_kmeans | Tidying methods for Spark ML unsupervised models
augment.ml_model_lda | Tidying methods for Spark ML LDA models
augment.ml_model_linear_regression | Tidying methods for Spark ML linear models
augment.ml_model_linear_svc | Tidying methods for Spark ML linear svc
augment.ml_model_logistic_regression | Tidying methods for Spark ML Logistic Regression
augment.ml_model_multilayer_perceptron_classification | Tidying methods for Spark ML MLP
augment.ml_model_naive_bayes | Tidying methods for Spark ML Naive Bayes
augment.ml_model_pca | Tidying methods for Spark ML Principal Component Analysis
augment.ml_model_random_forest_classification | Tidying methods for Spark ML tree models
augment.ml_model_random_forest_regression | Tidying methods for Spark ML tree models
augment._ml_model_decision_tree_classification | Tidying methods for Spark ML tree models
augment._ml_model_decision_tree_regression | Tidying methods for Spark ML tree models
augment._ml_model_gbt_classification | Tidying methods for Spark ML tree models
augment._ml_model_gbt_regression | Tidying methods for Spark ML tree models
augment._ml_model_linear_regression | Tidying methods for Spark ML linear models
augment._ml_model_logistic_regression | Tidying methods for Spark ML Logistic Regression
augment._ml_model_random_forest_classification | Tidying methods for Spark ML tree models
augment._ml_model_random_forest_regression | Tidying methods for Spark ML tree models

## -- C --

checkpoint_directory | Set/Get Spark checkpoint directory
collect_from_rds | Collect Spark data serialized in RDS format into R
compile_package_jars | Compile Scala sources into a Java Archive (jar)
connection_config | Read configuration values for a connection
copy_to.spark_connection | Copy an R Data Frame to Spark

## -- D --

distinct | Distinct
download_scalac | Downloads default Scala Compilers
dplyr_hof | dplyr wrappers for Apache Spark higher order functions

## -- E --

ensure | Enforce Specific Structure for R Objects

## -- F --

fill | Fill
filter | Filter
find_scalac | Discover the Scala Compiler
ft_binarizer | Feature Transformation - Binarizer (Transformer)
ft_bucketed_random_projection_lsh | Feature Transformation - LSH (Estimator)
ft_bucketizer | Feature Transformation - Bucketizer (Transformer)
ft_chisq_selector | Feature Transformation - ChiSqSelector (Estimator)
ft_count_vectorizer | Feature Transformation - CountVectorizer (Estimator)
ft_dct | Feature Transformation - Discrete Cosine Transform (DCT) (Transformer)
ft_discrete_cosine_transform | Feature Transformation - Discrete Cosine Transform (DCT) (Transformer)
ft_dplyr_transformer | Feature Transformation - SQLTransformer
ft_elementwise_product | Feature Transformation - ElementwiseProduct (Transformer)
ft_feature_hasher | Feature Transformation - FeatureHasher (Transformer)
ft_hashing_tf | Feature Transformation - HashingTF (Transformer)
ft_idf | Feature Transformation - IDF (Estimator)
ft_imputer | Feature Transformation - Imputer (Estimator)
ft_index_to_string | Feature Transformation - IndexToString (Transformer)
ft_interaction | Feature Transformation - Interaction (Transformer)
ft_lsh | Feature Transformation - LSH (Estimator)
ft_lsh_utils | Utility functions for LSH models
ft_max_abs_scaler | Feature Transformation - MaxAbsScaler (Estimator)
ft_minhash_lsh | Feature Transformation - LSH (Estimator)
ft_min_max_scaler | Feature Transformation - MinMaxScaler (Estimator)
ft_ngram | Feature Transformation - NGram (Transformer)
ft_normalizer | Feature Transformation - Normalizer (Transformer)
ft_one_hot_encoder | Feature Transformation - OneHotEncoder (Transformer)
ft_one_hot_encoder_estimator | Feature Transformation - OneHotEncoderEstimator (Estimator)
ft_pca | Feature Transformation - PCA (Estimator)
ft_polynomial_expansion | Feature Transformation - PolynomialExpansion (Transformer)
ft_quantile_discretizer | Feature Transformation - QuantileDiscretizer (Estimator)
ft_regex_tokenizer | Feature Transformation - RegexTokenizer (Transformer)
ft_robust_scaler | Feature Transformation - RobustScaler (Estimator)
ft_r_formula | Feature Transformation - RFormula (Estimator)
ft_sql_transformer | Feature Transformation - SQLTransformer
ft_standard_scaler | Feature Transformation - StandardScaler (Estimator)
ft_stop_words_remover | Feature Transformation - StopWordsRemover (Transformer)
ft_string_indexer | Feature Transformation - StringIndexer (Estimator)
ft_string_indexer_model | Feature Transformation - StringIndexer (Estimator)
ft_tokenizer | Feature Transformation - Tokenizer (Transformer)
ft_vector_assembler | Feature Transformation - VectorAssembler (Transformer)
ft_vector_indexer | Feature Transformation - VectorIndexer (Estimator)
ft_vector_slicer | Feature Transformation - VectorSlicer (Transformer)
ft_word2vec | Feature Transformation - Word2Vec (Estimator)
full_join | Full join
full_join.tbl_spark | Join Spark tbls.

## -- G --

generic_call_interface | Generic Call Interface
get_spark_sql_catalog_implementation | Retrieve the Spark connection's SQL catalog implementation property
glance.ml_model_aft_survival_regression | Tidying methods for Spark ML Survival Regression
glance.ml_model_als | Tidying methods for Spark ML ALS
glance.ml_model_bisecting_kmeans | Tidying methods for Spark ML unsupervised models
glance.ml_model_decision_tree_classification | Tidying methods for Spark ML tree models
glance.ml_model_decision_tree_regression | Tidying methods for Spark ML tree models
glance.ml_model_gaussian_mixture | Tidying methods for Spark ML unsupervised models
glance.ml_model_gbt_classification | Tidying methods for Spark ML tree models
glance.ml_model_gbt_regression | Tidying methods for Spark ML tree models
glance.ml_model_generalized_linear_regression | Tidying methods for Spark ML linear models
glance.ml_model_isotonic_regression | Tidying methods for Spark ML Isotonic Regression
glance.ml_model_kmeans | Tidying methods for Spark ML unsupervised models
glance.ml_model_lda | Tidying methods for Spark ML LDA models
glance.ml_model_linear_regression | Tidying methods for Spark ML linear models
glance.ml_model_linear_svc | Tidying methods for Spark ML linear svc
glance.ml_model_logistic_regression | Tidying methods for Spark ML Logistic Regression
glance.ml_model_multilayer_perceptron_classification | Tidying methods for Spark ML MLP
glance.ml_model_naive_bayes | Tidying methods for Spark ML Naive Bayes
glance.ml_model_pca | Tidying methods for Spark ML Principal Component Analysis
glance.ml_model_random_forest_classification | Tidying methods for Spark ML tree models
glance.ml_model_random_forest_regression | Tidying methods for Spark ML tree models

## -- H --

hive_context | Access the Spark API
hive_context_config | Runtime configuration interface for Hive
hof_aggregate | Apply Aggregate Function to Array Column
hof_array_sort | Sorts array using a custom comparator
hof_exists | Determine Whether Some Element Exists in an Array Column
hof_filter | Filter Array Column
hof_forall | Checks whether all elements in an array satisfy a predicate
hof_map_filter | Filters a map
hof_map_zip_with | Merges two maps into one
hof_transform | Transform Array Column
hof_transform_keys | Transforms keys of a map
hof_transform_values | Transforms values of a map
hof_zip_with | Combines 2 Array Columns

## -- I --

inner_join | Inner join
inner_join.tbl_spark | Join Spark tbls.
invoke | Invoke a Method on a JVM Object
invoke_new | Invoke a Method on a JVM Object
invoke_static | Invoke a Method on a JVM Object
is_ml_estimator | Spark ML - Transform, fit, and predict methods (ml_ interface)
is_ml_transformer | Spark ML - Transform, fit, and predict methods (ml_ interface)

## -- J --

jarray | Instantiate a Java array with a specific element type.
java_context | Access the Spark API
jfloat | Instantiate a Java float type.
jfloat_array | Instantiate an Array[Float].
join.tbl_spark | Join Spark tbls.
j_invoke | Invoke a Java function.
j_invoke_new | Invoke a Java function.
j_invoke_static | Invoke a Java function.

## -- L --

left_join | Left join
left_join.tbl_spark | Join Spark tbls.
list_sparklyr_jars | list all sparklyr-*.jar files that have been built
livy_config | Create a Spark Configuration for Livy
livy_service_start | Start Livy
livy_service_stop | Start Livy

## -- M --

ml-params | Spark ML - ML Params
ml-persistence | Spark ML - Model Persistence
ml-transform-methods | Spark ML - Transform, fit, and predict methods (ml_ interface)
ml-tuning | Spark ML - Tuning
ml_aft_survival_regression | Spark ML - Survival Regression
ml_als | Spark ML - ALS
ml_als_tidiers | Tidying methods for Spark ML ALS
ml_approx_nearest_neighbors | Utility functions for LSH models
ml_approx_similarity_join | Utility functions for LSH models
ml_association_rules | Frequent Pattern Mining - FPGrowth
ml_binary_classification_eval | Spark ML - Evaluators
ml_binary_classification_evaluator | Spark ML - Evaluators
ml_bisecting_kmeans | Spark ML - Bisecting K-Means Clustering
ml_chisquare_test | Chi-square hypothesis testing for categorical data.
ml_classification_eval | Spark ML - Evaluators
ml_clustering_evaluator | Spark ML - Clustering Evaluator
ml_compute_cost | Spark ML - K-Means Clustering
ml_compute_silhouette_measure | Spark ML - K-Means Clustering
ml_corr | Compute correlation matrix
ml_cross_validator | Spark ML - Tuning
ml_decision_tree | Spark ML - Decision Trees
ml_decision_tree_classifier | Spark ML - Decision Trees
ml_decision_tree_regressor | Spark ML - Decision Trees
ml_default_stop_words | Default stop words
ml_describe_topics | Spark ML - Latent Dirichlet Allocation
ml_evaluate | Evaluate the Model on a Validation Set
ml_evaluate.ml_evaluator | Evaluate the Model on a Validation Set
ml_evaluate.ml_generalized_linear_regression_model | Evaluate the Model on a Validation Set
ml_evaluate.ml_linear_regression_model | Evaluate the Model on a Validation Set
ml_evaluate.ml_logistic_regression_model | Evaluate the Model on a Validation Set
ml_evaluate.ml_model_classification | Evaluate the Model on a Validation Set
ml_evaluate.ml_model_clustering | Evaluate the Model on a Validation Set
ml_evaluate.ml_model_generalized_linear_regression | Evaluate the Model on a Validation Set
ml_evaluate.ml_model_linear_regression | Evaluate the Model on a Validation Set
ml_evaluate.ml_model_logistic_regression | Evaluate the Model on a Validation Set
ml_evaluator | Spark ML - Evaluators
ml_feature_importances | Spark ML - Feature Importance for Tree Models
ml_find_synonyms | Feature Transformation - Word2Vec (Estimator)
ml_fit | Spark ML - Transform, fit, and predict methods (ml_ interface)
ml_fit.default | Spark ML - Transform, fit, and predict methods (ml_ interface)
ml_fit_and_transform | Spark ML - Transform, fit, and predict methods (ml_ interface)
ml_fpgrowth | Frequent Pattern Mining - FPGrowth
ml_freq_itemsets | Frequent Pattern Mining - FPGrowth
ml_freq_seq_patterns | Frequent Pattern Mining - PrefixSpan
ml_gaussian_mixture | Spark ML - Gaussian Mixture clustering.
ml_gbt_classifier | Spark ML - Gradient Boosted Trees
ml_gbt_regressor | Spark ML - Gradient Boosted Trees
ml_generalized_linear_regression | Spark ML - Generalized Linear Regression
ml_glm_tidiers | Tidying methods for Spark ML linear models
ml_gradient_boosted_trees | Spark ML - Gradient Boosted Trees
ml_isotonic_regression | Spark ML - Isotonic Regression
ml_isotonic_regression_tidiers | Tidying methods for Spark ML Isotonic Regression
ml_is_set | Spark ML - ML Params
ml_kmeans | Spark ML - K-Means Clustering
ml_kmeans_cluster_eval | Evaluate a K-mean clustering
ml_labels | Feature Transformation - StringIndexer (Estimator)
ml_lda | Spark ML - Latent Dirichlet Allocation
ml_lda_tidiers | Tidying methods for Spark ML LDA models
ml_linear_regression | Spark ML - Linear Regression
ml_linear_svc | Spark ML - LinearSVC
ml_linear_svc_tidiers | Tidying methods for Spark ML linear svc
ml_load | Spark ML - Model Persistence
ml_logistic_regression | Spark ML - Logistic Regression
ml_logistic_regression_tidiers | Tidying methods for Spark ML Logistic Regression
ml_log_likelihood | Spark ML - Latent Dirichlet Allocation
ml_log_perplexity | Spark ML - Latent Dirichlet Allocation
ml_metrics_binary | Extracts metrics from a fitted table
ml_metrics_multiclass | Extracts metrics from a fitted table
ml_metrics_regression | Extracts metrics from a fitted table
ml_model_data | Extracts data associated with a Spark ML model
ml_multiclass_classification_evaluator | Spark ML - Evaluators
ml_multilayer_perceptron | Spark ML - Multilayer Perceptron
ml_multilayer_perceptron_classifier | Spark ML - Multilayer Perceptron
ml_multilayer_perceptron_tidiers | Tidying methods for Spark ML MLP
ml_naive_bayes | Spark ML - Naive-Bayes
ml_naive_bayes_tidiers | Tidying methods for Spark ML Naive Bayes
ml_one_vs_rest | Spark ML - OneVsRest
ml_param | Spark ML - ML Params
ml_params | Spark ML - ML Params
ml_param_map | Spark ML - ML Params
ml_pca | Feature Transformation - PCA (Estimator)
ml_pca_tidiers | Tidying methods for Spark ML Principal Component Analysis
ml_pipeline | Spark ML - Pipelines
ml_power_iteration | Spark ML - Power Iteration Clustering
ml_predict | Spark ML - Transform, fit, and predict methods (ml_ interface)
ml_predict.ml_model_classification | Spark ML - Transform, fit, and predict methods (ml_ interface)
ml_prefixspan | Frequent Pattern Mining - PrefixSpan
ml_random_forest | Spark ML - Random Forest
ml_random_forest_classifier | Spark ML - Random Forest
ml_random_forest_regressor | Spark ML - Random Forest
ml_recommend | Spark ML - ALS
ml_regression_evaluator | Spark ML - Evaluators
ml_save | Spark ML - Model Persistence
ml_save.ml_model | Spark ML - Model Persistence
ml_stage | Spark ML - Pipeline stage extraction
ml_stages | Spark ML - Pipeline stage extraction
ml_sub_models | Spark ML - Tuning
ml_summary | Spark ML - Extraction of summary metrics
ml_survival_regression | Spark ML - Survival Regression
ml_survival_regression_tidiers | Tidying methods for Spark ML Survival Regression
ml_topics_matrix | Spark ML - Latent Dirichlet Allocation
ml_train_validation_split | Spark ML - Tuning
ml_transform | Spark ML - Transform, fit, and predict methods (ml_ interface)
ml_tree_feature_importance | Spark ML - Feature Importance for Tree Models
ml_tree_tidiers | Tidying methods for Spark ML tree models
ml_uid | Spark ML - UID
ml_unsupervised_tidiers | Tidying methods for Spark ML unsupervised models
ml_validation_metrics | Spark ML - Tuning
ml_vocabulary | Feature Transformation - CountVectorizer (Estimator)
mutate | Mutate

## -- N --

na.replace | Replace Missing Values in Objects
nest | Nest

## -- P --

pivot_longer | Pivot longer
pivot_wider | Pivot wider

## -- R --

random_string | Random string generation
reactiveSpark | Reactive spark reader
registerDoSpark | Register a Parallel Backend
registered_extensions | Register a Package that Implements a Spark Extension
register_extension | Register a Package that Implements a Spark Extension
replace_na | Replace NA
right_join | Right join
right_join.tbl_spark | Join Spark tbls.

## -- S --

sdf-saveload | Save / Load a Spark DataFrame
sdf-transform-methods | Spark ML - Transform, fit, and predict methods (sdf_ interface)
sdf_along | Create DataFrame for along Object
sdf_bind | Bind multiple Spark DataFrames by row and column
sdf_bind_cols | Bind multiple Spark DataFrames by row and column
sdf_bind_rows | Bind multiple Spark DataFrames by row and column
sdf_broadcast | Broadcast hint
sdf_checkpoint | Checkpoint a Spark DataFrame
sdf_coalesce | Coalesces a Spark DataFrame
sdf_collect | Collect a Spark DataFrame into R.
sdf_copy_to | Copy an Object into Spark
sdf_crosstab | Cross Tabulation
sdf_debug_string | Debug Info for Spark DataFrame
sdf_describe | Compute summary statistics for columns of a data frame
sdf_dim | Support for Dimension Operations
sdf_distinct | Invoke distinct on a Spark DataFrame
sdf_drop_duplicates | Remove duplicates from a Spark DataFrame
sdf_expand_grid | Create a Spark dataframe containing all combinations of inputs
sdf_fit | Spark ML - Transform, fit, and predict methods (sdf_ interface)
sdf_fit_and_transform | Spark ML - Transform, fit, and predict methods (sdf_ interface)
sdf_from_avro | Convert column(s) from avro format
sdf_import | Copy an Object into Spark
sdf_is_streaming | Spark DataFrame is Streaming
sdf_last_index | Returns the last index of a Spark DataFrame
sdf_len | Create DataFrame for Length
sdf_load_parquet | Save / Load a Spark DataFrame
sdf_load_table | Save / Load a Spark DataFrame
sdf_ncol | Support for Dimension Operations
sdf_nrow | Support for Dimension Operations
sdf_num_partitions | Gets number of partitions of a Spark DataFrame
sdf_partition | Partition a Spark Dataframe
sdf_partition_sizes | Compute the number of records within each partition of a Spark DataFrame
sdf_persist | Persist a Spark DataFrame
sdf_pivot | Pivot a Spark DataFrame
sdf_predict | Spark ML - Transform, fit, and predict methods (sdf_ interface)
sdf_project | Project features onto principal components
sdf_quantile | Compute (Approximate) Quantiles with a Spark DataFrame
sdf_random_split | Partition a Spark Dataframe
sdf_rbeta | Generate random samples from a Beta distribution
sdf_rbinom | Generate random samples from a binomial distribution
sdf_rcauchy | Generate random samples from a Cauchy distribution
sdf_rchisq | Generate random samples from a chi-squared distribution
sdf_read_column | Read a Column from a Spark DataFrame
sdf_register | Register a Spark DataFrame
sdf_repartition | Repartition a Spark DataFrame
sdf_residuals | Model Residuals
sdf_residuals.ml_model_generalized_linear_regression | Model Residuals
sdf_residuals.ml_model_linear_regression | Model Residuals
sdf_rexp | Generate random samples from an exponential distribution
sdf_rgamma | Generate random samples from a Gamma distribution
sdf_rgeom | Generate random samples from a geometric distribution
sdf_rhyper | Generate random samples from a hypergeometric distribution
sdf_rlnorm | Generate random samples from a log normal distribution
sdf_rnorm | Generate random samples from the standard normal distribution
sdf_rpois | Generate random samples from a Poisson distribution
sdf_rt | Generate random samples from a t-distribution
sdf_runif | Generate random samples from the uniform distribution U(0, 1).
sdf_rweibull | Generate random samples from a Weibull distribution.
sdf_sample | Randomly Sample Rows from a Spark DataFrame
sdf_save_parquet | Save / Load a Spark DataFrame
sdf_save_table | Save / Load a Spark DataFrame
sdf_schema | Read the Schema of a Spark DataFrame
sdf_separate_column | Separate a Vector Column into Scalar Columns
sdf_seq | Create DataFrame for Range
sdf_sort | Sort a Spark DataFrame
sdf_sql | Spark DataFrame from SQL
sdf_to_avro | Convert column(s) to avro format
sdf_transform | Spark ML - Transform, fit, and predict methods (sdf_ interface)
sdf_unnest_longer | Unnest longer
sdf_unnest_wider | Unnest wider
sdf_weighted_sample | Perform Weighted Random Sampling on a Spark DataFrame
sdf_with_sequential_id | Add a Sequential ID Column to a Spark DataFrame
sdf_with_unique_id | Add a Unique ID Column to a Spark DataFrame
select | Select
separate | Separate
spark-api | Access the Spark API
spark-connections | Manage Spark Connections
sparklyr_get_backend_port | Return the port number of a 'sparklyr' backend.
spark_adaptive_query_execution | Retrieves or sets status of Spark AQE
spark_advisory_shuffle_partition_size | Retrieves or sets advisory size of the shuffle partition
spark_apply | Apply an R Function in Spark
spark_apply_bundle | Create Bundle for Spark Apply
spark_apply_log | Log Writer for Spark Apply
spark_auto_broadcast_join_threshold | Retrieves or sets the auto broadcast join threshold
spark_available_versions | Download and install various versions of Spark
spark_coalesce_initial_num_partitions | Retrieves or sets initial number of shuffle partitions before coalescing
spark_coalesce_min_num_partitions | Retrieves or sets the minimum number of shuffle partitions after coalescing
spark_coalesce_shuffle_partitions | Retrieves or sets whether coalescing contiguous shuffle partitions is enabled
spark_compilation_spec | Define a Spark Compilation Specification
spark_config | Read Spark Configuration
spark_config_kubernetes | Kubernetes Configuration
spark_config_settings | Retrieve Available Settings
spark_connect | Manage Spark Connections
spark_connection | Retrieve the Spark Connection Associated with an R Object
spark_connection-class | spark_connection class
spark_connection_find | Find Spark Connection
spark_connection_is_open | Manage Spark Connections
spark_connect_method | Function that negotiates the connection with the Spark back-end
spark_context | Access the Spark API
spark_context_config | Runtime configuration interface for the Spark Context.
spark_dataframe | Retrieve a Spark DataFrame
spark_default_compilation_spec | Default Compilation Specification for Spark Extensions
spark_dependency | Define a Spark dependency
spark_dependency_fallback | Fallback to Spark Dependency
spark_disconnect | Manage Spark Connections
spark_disconnect_all | Manage Spark Connections
spark_extension | Create Spark Extension
spark_get_checkpoint_dir | Set/Get Spark checkpoint directory
spark_home_set | Set the SPARK_HOME environment variable
spark_ide_columns | Set of functions to provide integration with the RStudio IDE
spark_ide_connection_actions | Set of functions to provide integration with the RStudio IDE
spark_ide_connection_closed | Set of functions to provide integration with the RStudio IDE
spark_ide_connection_open | Set of functions to provide integration with the RStudio IDE
spark_ide_connection_updated | Set of functions to provide integration with the RStudio IDE
spark_ide_objects | Set of functions to provide integration with the RStudio IDE
spark_ide_preview | Set of functions to provide integration with the RStudio IDE
spark_insert_table | Inserts a Spark DataFrame into a Spark table
spark_install | Download and install various versions of Spark
spark_installed_versions | Download and install various versions of Spark
spark_install_dir | Download and install various versions of Spark
spark_install_tar | Download and install various versions of Spark
spark_integ_test_skip | It lets the package know if it should test a particular functionality or not
spark_jobj | Retrieve a Spark JVM Object Reference
spark_jobj-class | spark_jobj class
spark_last_error | Surfaces the last error from Spark captured by internal 'spark_error' function
spark_load_table | Reads from a Spark Table into a Spark DataFrame.
spark_log | View Entries in the Spark Log
spark_read | Read file(s) into a Spark DataFrame using a custom reader
spark_read_avro | Read Apache Avro data into a Spark DataFrame.
spark_read_binary | Read binary data into a Spark DataFrame.
spark_read_csv | Read a CSV file into a Spark DataFrame
spark_read_delta | Read from Delta Lake into a Spark DataFrame.
spark_read_image | Read image data into a Spark DataFrame.
spark_read_jdbc | Read from JDBC connection into a Spark DataFrame.
spark_read_json | Read a JSON file into a Spark DataFrame
spark_read_libsvm | Read libsvm file into a Spark DataFrame.
spark_read_orc | Read a ORC file into a Spark DataFrame
spark_read_parquet | Read a Parquet file into a Spark DataFrame
spark_read_source | Read from a generic source into a Spark DataFrame.
spark_read_table | Reads from a Spark Table into a Spark DataFrame.
spark_read_text | Read a Text file into a Spark DataFrame
spark_save_table | Saves a Spark DataFrame as a Spark table
spark_session | Access the Spark API
spark_session_config | Runtime configuration interface for the Spark Session
spark_set_checkpoint_dir | Set/Get Spark checkpoint directory
spark_statistical_routines | Generate random samples from some distribution
spark_submit | Manage Spark Connections
spark_table_name | Generate a Table Name from Expression
spark_uninstall | Download and install various versions of Spark
spark_version | Get the Spark Version Associated with a Spark Connection
spark_version_from_home | Get the Spark Version Associated with a Spark Installation
spark_web | Open the Spark web interface
spark_write | Write Spark DataFrame to file using a custom writer
spark_write_avro | Serialize a Spark DataFrame into Apache Avro format
spark_write_csv | Write a Spark DataFrame to a CSV
spark_write_delta | Writes a Spark DataFrame into Delta Lake
spark_write_jdbc | Writes a Spark DataFrame into a JDBC table
spark_write_json | Write a Spark DataFrame to a JSON file
spark_write_orc | Write a Spark DataFrame to a ORC file
spark_write_parquet | Write a Spark DataFrame to a Parquet file
spark_write_rds | Write Spark DataFrame to RDS files
spark_write_source | Writes a Spark DataFrame into a generic source
spark_write_table | Writes a Spark DataFrame into a Spark table
spark_write_text | Write a Spark DataFrame to a Text file
src_databases | Show database list
stream_find | Find Stream
stream_generate_test | Generate Test Stream
stream_id | Spark Stream's Identifier
stream_lag | Apply lag function to columns of a Spark Streaming DataFrame
stream_name | Spark Stream's Name
stream_read_csv | Read files created by the stream
stream_read_delta | Read files created by the stream
stream_read_json | Read files created by the stream
stream_read_kafka | Read files created by the stream
stream_read_orc | Read files created by the stream
stream_read_parquet | Read files created by the stream
stream_read_socket | Read files created by the stream
stream_read_text | Read files created by the stream
stream_render | Render Stream
stream_stats | Stream Statistics
stream_stop | Stops a Spark Stream
stream_trigger_continuous | Spark Stream Continuous Trigger
stream_trigger_interval | Spark Stream Interval Trigger
stream_view | View Stream
stream_watermark | Watermark Stream
stream_write_console | Write files to the stream
stream_write_csv | Write files to the stream
stream_write_delta | Write files to the stream
stream_write_json | Write files to the stream
stream_write_kafka | Write files to the stream
stream_write_memory | Write Memory Stream
stream_write_orc | Write files to the stream
stream_write_parquet | Write files to the stream
stream_write_text | Write files to the stream

## -- T --

tbl_cache | Cache a Spark Table
tbl_change_db | Use specific database
tbl_uncache | Uncache a Spark Table
tidy.ml_model_aft_survival_regression | Tidying methods for Spark ML Survival Regression
tidy.ml_model_als | Tidying methods for Spark ML ALS
tidy.ml_model_bisecting_kmeans | Tidying methods for Spark ML unsupervised models
tidy.ml_model_decision_tree_classification | Tidying methods for Spark ML tree models
tidy.ml_model_decision_tree_regression | Tidying methods for Spark ML tree models
tidy.ml_model_gaussian_mixture | Tidying methods for Spark ML unsupervised models
tidy.ml_model_gbt_classification | Tidying methods for Spark ML tree models
tidy.ml_model_gbt_regression | Tidying methods for Spark ML tree models
tidy.ml_model_generalized_linear_regression | Tidying methods for Spark ML linear models
tidy.ml_model_isotonic_regression | Tidying methods for Spark ML Isotonic Regression
tidy.ml_model_kmeans | Tidying methods for Spark ML unsupervised models
tidy.ml_model_lda | Tidying methods for Spark ML LDA models
tidy.ml_model_linear_regression | Tidying methods for Spark ML linear models
tidy.ml_model_linear_svc | Tidying methods for Spark ML linear svc
tidy.ml_model_logistic_regression | Tidying methods for Spark ML Logistic Regression
tidy.ml_model_multilayer_perceptron_classification | Tidying methods for Spark ML MLP
tidy.ml_model_naive_bayes | Tidying methods for Spark ML Naive Bayes
tidy.ml_model_pca | Tidying methods for Spark ML Principal Component Analysis
tidy.ml_model_random_forest_classification | Tidying methods for Spark ML tree models
tidy.ml_model_random_forest_regression | Tidying methods for Spark ML tree models
transform_sdf | transform a subset of column(s) in a Spark Dataframe

## -- U --

unite | Unite
unnest | Unnest

## -- misc --

%->% | Infix operator for composing a lambda expression
[.tbl_spark | Subsetting operator for Spark dataframe