Let’s Write and Grow Together
I was never a topper.
I was never a failure either.
I lived somewhere in the “decent marks zone” — the middle child of academia. Not loud enough to be noticed, not quiet enough to be forgotten.
But if you asked me what I really lacked during my school days, it wasn’t grades — it was confidence.
In this post, we will see the list of questions asked with 3+ YOE candidate in Impetus Company Interview for Data Engineer profile.
Python is a fundamental language for Data Engineering, widely used in data processing, ETL pipelines, and big data frameworks like PySpark. To help you ace your Data Engineering interviews, I’m starting a Python Q&A series where we will cover commonly asked questions along with detailed explanations.
When working with data, you often encounter scenarios where a single column contains values that need to be split into multiple columns for easier analysis or processing. PySpark provides flexible way to achieve this using the split() function. In this article, we’ll cover how to split a single column into multiple columns in a PySpark […]
In this post, we will see the list of questions asked with 4+ YOE candidate in Tiger Analytics Company Interview for AWS Data Engineer profile.
Let’s see the Questions:
When working with data in PySpark, ensuring the correct data type for each column is essential for accurate analysis and processing. Sometimes, the data types of columns may not match your requirements. For example, a column containing numeric data might be stored as a string (string), or dates may be stored in an incorrect format.
When working with text data in Spark, you might come across special characters that don’t belong to the standard English alphabet. These characters are called non-ASCII characters. For example, accented letters like é in “José” or symbols like emojis ?. Sometimes, you may need to clean your data by removing these characters. This article will show you how to identify and remove non-ASCII characters from a Spark DataFrame.
Handling NULL (or None) values is a crucial task in data processing, as missing data can skew analysis, produce errors in data transformations, and degrade the performance of machine learning models. In PySpark, dealing with NULL values is a common operation when working with distributed datasets. PySpark provides several methods and techniques to detect, manage, and clean up missing or NULL values in a DataFrame.
In this blog post, we’ll explore how to handle NULL values in PySpark DataFrames, covering essential methods like filtering, filling, dropping, and replacing NULL values.
When working with large datasets in PySpark, it’s common to encounter duplicate records that can skew your analysis or cause issues in downstream processing. Fortunately, PySpark provides some methods to identify and remove duplicate rows from a DataFrame, ensuring that the data is clean and ready for analysis. In this article, we’ll explore two methods to remove duplicates from a PySpark DataFrame: dropDuplicates() and distinct().
Sorting data is a fundamental task in data processing, whether for analysis, reporting, or data transformation. In PySpark, sorting a DataFrame is a common operation that allows you to organize your data based on one or more columns. PySpark provides multiple ways to sort data efficiently, even when dealing with large datasets distributed across clusters.
In this blog post, we’ll explore various methods to sort a DataFrame in PySpark, covering both ascending and descending orders, sorting by multiple columns, and handling null values during sorting.
Connect 1:1 With Me: Schedule Call
If you have any doubts or would like to discuss anything related to this blog, feel free to reach out to me. I'm here to help! You can schedule a call by clicking on the above given link.
I'm looking forward to hearing from you and assisting you with any inquiries you may have. Your understanding and engagement are important to me!
This will close in 20 seconds