Видео

This is amazing, very well explained Sumit Sir! Do you give career advice also?

Spark core -Rdd (flexible) high level apis- Df and Spark sql (easy to write query) Transformation n action Spark submit process Deployment modes Types of transformation Repartition n coalesce Methods for schema enforcement - ddl, struct Consecutive wins in sql

Java used in Hadoop Bound to work on mapreduce Can only work on batch process not real time in map reduce

Data lake vs delta lake Unity catalog Data profiling Data governance

Predicate pushdown What opt used in spark Transformation used Groupby n reduceby Faced oom error ? Salting Data skewness Data spillness Cache persists Lru Repartition vs coalesce Rnk densernk rno What happen submit spark job

I definitely learnt something. Thanks.🎉

first one Select name from salary_table as a where salary>(select salary from salary_table where id =a.managerid)

What is the answer to the last question - 'When we align transformations and action in a DAG, what is that graph called?'

Dear Sir, hope you are doing well, when can we expect the next video on Python? It's been more than 3 months now, that you haven't posted any followed video on Python. I guess everyone is eagerly waiting. Please post it asap. Thanks

sir please upload more videos. We are waiting for the next

interview Series are good , please upload remaining 10 questions , eagerly waiting sir

Master and slave architecture we have driver node acts master node multiple worker nodes submitted spark jobs contexts of spark node - create jobs and execute plan spark cluster entry point driver will request resource manager send back to driver also do iteration of jobs

Great questions....

The interview series is really helpful, Thank you

One correctness here 9.59, in Narrow transformation there is no shuffling and in wide transformation there is shuffling of data.

Good content!

please share the link of these 175 questions of leetcode..it would be very helpful

Thanks for conducting these mock interviews Sumit sir. It is really helpful😊

Parquet is columnar based file format which stores the metadata along with the original data. i.e. MIN MAX values of the different columns in that file. During Read operation it checks the metadata and avoids scanning entire file that are irrelevant. Also by default it comes with Snappy compression which saves good amount of storage space.

🎯 Key points for quick navigation: 00:24 *🎓 Anur's background and expertise* - Anur has 9 years of industry experience, with over 7 years in Big Data. - Anur's primary skills revolve around Spark, AWS, cloud, data bricks, Kafka, and airflow. - Anur is currently working as an assistant manager at KPMG Global Services. 02:41 *🛠️ Ram's project overview (Insurance & Green Energy domains)* - Ram's previous project involved data injection, cleaning, transformation from raw to gold layer tables in a medallion architecture. - Ram's current project in the green energy domain includes moving data between systems, managing data warehouse, and creating a data strategy for the company. - Ram uses a variety of sources for data ingestion, from APIs to third-party sources dumped in ADLS Gen2. 05:29 *🔐 Data Security Framework Implementation* - Ram implemented an entitlement framework in Databricks for row-level and column-level masking. - Encryption methods were used, with keys stored in Azure Key Vault linked to Databricks using DBUtils secret scope. - Utilized Databricks Delta Lake for file format and ensured data security across various domains within the project. 11:44 *🔄 Resolving Pipeline Latency Bottlenecks* - Ram faced a bottleneck due to processing slowdown with accumulated data in a Databricks pipeline. - Resolved by optimizing cluster configurations, increasing CPU cores, and adjusting shuffle partitions for better performance. - Leveraged properties like cost-based optimization, join reordering, and adaptive query execution in Databricks for further pipeline optimization. 16:52 *📊 Real-time Dashboard Pipeline Implementation* - Ram's approach for a real-time dashboard involves capturing data from Oracle in a live stream, storing in Azure ADLS Gen2, and aggregating for the dashboard. - Explained the use of CDC feature, ADF integration runtime, and operations in Databricks for updating and reflecting real-time data changes on the dashboard. - Implemented triggers in Azure Data Factory, focusing on storage-based triggers to capture file arrivals for immediate processing. 21:29 *📊 Initial Spark optimization explanation* - Transformations in Spark are lazy and added to the directed acyclic graph (DAG) for evaluation when data is requested. - Spark applies lazy evaluation, leading to optimized action execution for processing only necessary data. 22:25 *🖥️ Cluster configuration and executor numbers* - Calculating ideal executor numbers and memory allocation based on data size and CPU cores. - Starting with a baseline of the number of executor cores for optimal cluster performance. 24:58 *💻 Approaching a Spark SQL problem* - Creating a cumulative revenue calculation from a DataFrame in Spark SQL. - Utilizing window functions and data manipulation to achieve the desired output. Made with HARPA AI

bhai ye toh reverse list k case me cheating kr rha..side me notes dekh kar answer likh rha..waah

are you doing on command prompt or what

Insert INTO emp2 Select * from emp1;

3rd scenario is not clear. Are we updating ant congratulation setting to avoid small files problem

What is cardinality

with c as(select e.*,m.salary as manager_salary,m.name as manager_name from employees_prac e left join employees_prac m on e.managerid=m.id where e.salary-m.salary>0) select name from c; My solution

Very helpful 🎉🎉🎉

LoL Why this question relevant ?

Dear sir I am facing an issue when giving mysql-ctl cli; which shows command not found... Kindly help

Thank you sir , this videos are very helpful

Sir when is next video coming?

Best video for starters <3

Will u teach urself or someone else?

Hi sumit sir . In this 5th video . The video description is without the topic list . Can you please add.

As per me, we can do the second question with below : with t1 as ( select customer_number,count(*) as count from orders group by customer_number) select t1.customer_number from t1 where t1.count = (select max(count) from t1)

Thank you! This is very helpful

Order by and distinct will work together if you add order by column name with select.

Nice videos

very nice way to teach. thank u Sir.

with cte as ( select *, row_number() over (order by start_time) as rownum, DATEADD(MINUTE,-1* row_number() over (order by start_time) ,start_time) as updated_time from service_status where status = 'down' ) select service_name,min(start_time) as start_updated_time,max(start_time) as end_updated_time, status from cte group by service_name,updated_time,status having count(*)>3

Clear and Great Explanation on Where and Having Clause 👌👌👌👌

Dono hi ase hn ,... Interviewer ko nind aa rhi h, aur candidate uska fayda uthana chahta h😂