When working with data, you often encounter scenarios where a single column contains values that need to be split into multiple columns for easier analysis or processing. PySpark provides flexible way to achieve this using the split() function. In this article, we’ll cover how to split a single column into multiple columns in a PySpark […]
Author: Ankit Rai
Tiger Analytics | Data Engineer Interview Questions – Set 1
In this post, we will see the list of questions asked with 4+ YOE candidate in Tiger Analytics Company Interview for AWS Data Engineer profile.
Let’s see the Questions:
PySpark | How to Perform Data Type Casting on Columns in a DataFrame?
When working with data in PySpark, ensuring the correct data type for each column is essential for accurate analysis and processing. Sometimes, the data types of columns may not match your requirements. For example, a column containing numeric data might be stored as a string (string), or dates may be stored in an incorrect format.
PySpark | How to Remove Non-ASCII Characters from a DataFrame?
When working with text data in Spark, you might come across special characters that don’t belong to the standard English alphabet. These characters are called non-ASCII characters. For example, accented letters like é in “José” or symbols like emojis 😊. Sometimes, you may need to clean your data by removing these characters. This article will show you how to identify and remove non-ASCII characters from a Spark DataFrame.
PySpark | How to Handle Nulls in DataFrame?
Handling NULL (or None) values is a crucial task in data processing, as missing data can skew analysis, produce errors in data transformations, and degrade the performance of machine learning models. In PySpark, dealing with NULL values is a common operation when working with distributed datasets. PySpark provides several methods and techniques to detect, manage, and clean up missing or NULL values in a DataFrame.
In this blog post, we’ll explore how to handle NULL values in PySpark DataFrames, covering essential methods like filtering, filling, dropping, and replacing NULL values.
PySpark | How to remove duplicates from Dataframe?
When working with large datasets in PySpark, it’s common to encounter duplicate records that can skew your analysis or cause issues in downstream processing. Fortunately, PySpark provides some methods to identify and remove duplicate rows from a DataFrame, ensuring that the data is clean and ready for analysis. In this article, we’ll explore two methods to remove duplicates from a PySpark DataFrame: dropDuplicates() and distinct().
PySpark | How to Sort a Dataframe?
Sorting data is a fundamental task in data processing, whether for analysis, reporting, or data transformation. In PySpark, sorting a DataFrame is a common operation that allows you to organize your data based on one or more columns. PySpark provides multiple ways to sort data efficiently, even when dealing with large datasets distributed across clusters.
In this blog post, we’ll explore various methods to sort a DataFrame in PySpark, covering both ascending and descending orders, sorting by multiple columns, and handling null values during sorting.
Python Tutorial | Learn Python Programming
Python is a versatile and beginner-friendly programming language that has gained immense popularity for its simplicity, readability, and wide range of applications. Whether you’re new to programming or looking to expand your skills, learning Python is an excellent choice. In this comprehensive guide, i’ll provide you with a curated list of resources and tutorials from my website to help you master Python programming from scratch.
PySpark Tutorial | Learn PySpark
PySpark is the Python API for Apache Spark, a powerful open-source framework designed for distributed computing and processing large datasets. By combining the scalability and performance of Spark with Python’s simplicity, PySpark has become an essential tool for data engineers and data scientists working with big data.
PySpark | How to Filter Data in DataFrame?
Filtering data is one of the most common operations you’ll perform when working with PySpark DataFrames. Whether you’re analyzing large datasets, preparing data for machine learning models, or performing transformations, you often need to isolate specific subsets of data based on certain conditions. PySpark provides several methods for filtering DataFrames, and this article will explore the most widely used approaches.