SelfReflex
SelfReflex
  • 52
  • 143 230

Відео

Apache Spark Tutorial: SQL like Functions on Datasets - select, filter, groupBy - hands on
Переглядів 78Рік тому
Welcome to the immersive 13th session of our comprehensive Apache Spark tutorial series! In this session, we dive deep into the world of hands-on exercises, focusing on SQL-like functions on Datasets. Get ready to master the art of data manipulation in Apache Spark! In this session, we'll guide you through a series of practical exercises that harness the power of SQL-like functions on Datasets....
Apache Spark Tutorial: Hands-on Spark SQL, Datasets, and DataFrames: Dataset vs Dataframe
Переглядів 96Рік тому
Welcome to the exhilarating 12th session of our comprehensive Apache Spark tutorial series! In this session, we dive straight into the world of hands-on exercises with Spark SQL, Datasets, and DataFrames, taking you on an immersive journey of practical structured data processing in Apache Spark. In this session, we roll up our sleeves and embark on a series of hands-on exercises designed to sol...
Apache Spark Tutorial: Introduction To Spark SQL, Datasets, and DataFrames
Переглядів 108Рік тому
Welcome to the captivating 11th video of our comprehensive Apache Spark tutorial series! In this session, we dive into the world of Spark SQL, Datasets, and DataFrames, where we unlock the full potential of structured data processing in Apache Spark. In this session, we take you on a journey through the rich capabilities of Spark SQL, the module that enables SQL-like querying and processing of ...
Apache Spark Tutorial: Mastering Jobs, Stages, Tasks, and Spark's logical execution plan
Переглядів 667Рік тому
Welcome to the captivating tenth session of our comprehensive Apache Spark tutorial series! In this session, we delve into the core concepts of Spark's data processing engine, where we demystify the intricate relationship between jobs, stages, tasks, and the all-important shuffle sort operation. In this session, we provide a comprehensive overview of how Spark organizes and executes data proces...
Apache Spark Tutorial: Deep Dive into Apache Spark Runtime Architecture on YARN - Spark Internals
Переглядів 116Рік тому
Welcome to the enlightening ninth session of our comprehensive Apache Spark tutorial series! In this session, we embark on a fascinating exploration of the Spark runtime architecture on YARN (Yet Another Resource Negotiator). Join us as we peel back the layers and unravel the inner workings of Spark's execution framework in a YARN cluster environment. In this session, we provide an in-depth ove...
Apache Spark Tutorial: Broadcast Variables Hands-On Exercise: Real-Life Use Case From Telecom World
Переглядів 83Рік тому
Welcome to the highly practical eighth session of our comprehensive Apache Spark tutorial series! In this session, we delve into the powerful world of broadcast variables and guide you through a hands-on exercise using a real-life use case from the telecom industry. Join us as we explore how broadcast variables can optimize data sharing in large-scale telecom data processing and analytics. In t...
Apache Spark Tutorial: Deep Dive into Broadcast Variables in Apache Spark - Unveiling the Internals
Переглядів 75Рік тому
Welcome to the captivating seventh session of our comprehensive Apache Spark tutorial series! In this session, we embark on an exciting journey into the inner workings of broadcast variables in Spark. Join us as we unravel the mysteries behind broadcast variables, exploring how they optimize data sharing and improve performance in distributed computations. We dive into the internals, shedding l...
Apache Spark Tutorial: map vs flatmap
Переглядів 126Рік тому
Welcome to the highly informative sixth session of our comprehensive Apache Spark tutorial series! In this session, we delve into the essential concepts of the Map and FlatMap operations in Spark, highlighting the key differences between them. Join us as we unravel the nuances of Map and FlatMap and explore when to use each operation effectively. By the end of this session, you'll have a solid ...
Apache Spark Tutorial: RDD Filter Operation Explained with Hands-on Exercise for Data Filtering
Переглядів 126Рік тому
Welcome to the highly anticipated fifth session of our comprehensive Apache Spark tutorial series! In this exciting session, we deep-dive into the RDD filter operation and its powerful capabilities for data filtering in Spark. Join us as we demystify the filter operation through a hands-on exercise that will sharpen your skills in data filtering and transformation. We'll guide you step-by-step ...
Apache Spark Tutorial: Exploring Key-Value RDDs with Hands-on Exercise for Big Data Processing
Переглядів 116Рік тому
Welcome to the highly anticipated fourth session of our comprehensive Apache Spark tutorial series! In this dynamic session, we delve into the intricacies of Key-Value RDDs (Resilient Distributed Datasets) and demonstrate their immense value in advanced big data processing. Join us as we uncover the power of Key-Value RDDs through hands-on exercises and practical examples. Through this immersiv...
How to decide number of executors | Apache Spark Interview Questions
Переглядів 116Рік тому
How to decide number of executors for a spark job? AN important interview question for apache spark. watch to find out the answer.
When does Apache Spark compute RDD Lineage | Apache Spark Interview Questions with answers
Переглядів 34Рік тому
When does Apache Spark compute RDD Lineage? watch to figure out how to answer this interview question on apache spark.
Is Spark better than Hadoop? | Apache Spark Interview Questions | With Explanation
Переглядів 47Рік тому
Apache spark interview question. Is spark better than hadoop? watch to know the answer. If you think you know the answer, you might be wrong. If you think the question is wrong, watch to find out.
Apache Spark Tutorial: Hands-on Exercise with RDDs for Big Data Processing | Learning RDDs
Переглядів 179Рік тому
Welcome back to our engaging Apache Spark tutorial series! In this exciting third session, we roll up our sleeves and dive into a hands-on exercise with RDDs (Resilient Distributed Datasets) to unleash the full potential of Spark for big data processing. Join us as we guide you through a step-by-step practical exercise, where you'll gain invaluable experience in manipulating RDDs, applying tran...
Apache Spark Tutorial: Mastering RDDs (Resilient Distributed Datasets) for Big Data Processing
Переглядів 231Рік тому
Apache Spark Tutorial: Mastering RDDs (Resilient Distributed Datasets) for Big Data Processing
Apache Spark Tutorial: Introduction and Basics of Big Data Processing
Переглядів 594Рік тому
Apache Spark Tutorial: Introduction and Basics of Big Data Processing
Introduction To Kubernetes
Переглядів 20Рік тому
Introduction To Kubernetes
Yarn architecture introduction
Переглядів 7 тис.4 роки тому
Yarn architecture introduction
SelfReflex Big Data Architecture
Переглядів 2837 років тому
SelfReflex Big Data Architecture
Master Map Reduce in 2 hours
Переглядів 1,4 тис.7 років тому
Master Map Reduce in 2 hours
Hadoop interview questions Explain map reduce paradigm with example
Переглядів 1,2 тис.7 років тому
Hadoop interview questions Explain map reduce paradigm with example
Hadoop interview questions map reduce vs traditional data processing
Переглядів 8137 років тому
Hadoop interview questions map reduce vs traditional data processing
Hadoop interview questions what is map reduce
Переглядів 1,2 тис.7 років тому
Hadoop interview questions what is map reduce
Master YARN in 45 minutes
Переглядів 29 тис.7 років тому
Master YARN in 45 minutes
Hadoop interview questions hadoop 1 0 vs hadoop 2 0
Переглядів 9657 років тому
Hadoop interview questions hadoop 1 0 vs hadoop 2 0
Hadoop interview questions reason for write failures
Переглядів 1 тис.7 років тому
Hadoop interview questions reason for write failures
Hadoop interview questions secondary vs standby namenode
Переглядів 1,5 тис.7 років тому
Hadoop interview questions secondary vs standby namenode
Hadoop interview questions hdfs node failures
Переглядів 1,3 тис.7 років тому
Hadoop interview questions hdfs node failures
hadoop interview questions what is rack awareness
Переглядів 6 тис.7 років тому
hadoop interview questions what is rack awareness

КОМЕНТАРІ

  • @arzanishyusufnadeem2086
    @arzanishyusufnadeem2086 3 місяці тому

    Great explanation !

  • @Anonymous-fo2rv
    @Anonymous-fo2rv 8 місяців тому

    best explanation not only on youtube but the entire internet .............Thank you so much

  • @krishshah8881
    @krishshah8881 Рік тому

    after watching lots of videos and getting more confused this video showed up and unexpectedly turned out to be a life saver hats off to you sir

  • @javedabdool1187
    @javedabdool1187 Рік тому

    do you have a simple code for the traditional way or another more explicit video?

  • @shekharraghuvanshi2267
    @shekharraghuvanshi2267 Рік тому

    This 45 minutes lecture is a master piece..!!! Nothing else is even close to this one.

  • @MrPrince
    @MrPrince Рік тому

    Thanks for the amazing explanations Omar from Egypt :)

    • @Selfreflex
      @Selfreflex Рік тому

      Glad it was helpful!

    • @MrPrince
      @MrPrince Рік тому

      @@Selfreflex It was helpful indeed. I got an A+ in my final exams in big data 🤓

    • @Selfreflex
      @Selfreflex Рік тому

      @@MrPrince keep it up! Looking forward to serve more content for the learners like you!

  • @shoibsiddiqui6954
    @shoibsiddiqui6954 Рік тому

    Hi can you share your contact....

  • @funnysatisfying426
    @funnysatisfying426 Рік тому

    One slave node can have more than one container right?

  • @shubhamrout992
    @shubhamrout992 Рік тому

    Consider a retail shop with 10 employees where each employees gets a monthly bonous of he number of sales select * from employee distributed by employee_id sorted by sales_amount If the above command executed, how many reducer will be used ?? A)Canot be determined B)1.0 C)no reducer will be used D)10 Kindly ans my questions please

  • @josephsagi3926
    @josephsagi3926 Рік тому

    Before coming to this video, I wasted 2 days searching many websites and videos for understanding YARN..! I hesitated to take video b/z of time but I AM LUCKY ALL MY DOUBTS GOT CLEARED ON YARN..! THANKS.. 👍🙏

  • @kancharlakeerthi1819
    @kancharlakeerthi1819 2 роки тому

    thank you so much its a clear explanation

  • @dapomaster6089
    @dapomaster6089 2 роки тому

    Nicely explained sir

  • @nikhilpatil1958
    @nikhilpatil1958 2 роки тому

    sir can you share this ppt with us

  • @rajarajan9509
    @rajarajan9509 2 роки тому

    Awesome sir. Thank you for giving this. Your website is not working .plz upload many videos like this....

  • @happymommy4693
    @happymommy4693 2 роки тому

    U made it easy

  • @Manishkrpandey97
    @Manishkrpandey97 2 роки тому

    According to your explanation it looks like no. of mapper is equal to no. of block size ...but generally no. of mapper is equal to no. of input split..pls let me know if i am wrong 🤔

  • @shahezadalam6709
    @shahezadalam6709 2 роки тому

    greate job

  • @williamhaque6183
    @williamhaque6183 2 роки тому

    Well explained. Great job. Thank you.

  • @dastan285
    @dastan285 2 роки тому

    Secondary namenode is standby namenode

  • @IrfanShaikh-jt6yu
    @IrfanShaikh-jt6yu 3 роки тому

    before coming here i searched all the blogs about yarn but didnt satisfied but this 45min cleared all the concepts even how actual data process i understood.Thank you sir,awesome explanation,VERY USEFULL

  • @allisavd
    @allisavd 3 роки тому

    Very nicely explained the concept of split size.

  • @Dragoncreativelabs
    @Dragoncreativelabs 3 роки тому

    Totally worth it, a clean explanation!

  • @sumitayadav9407
    @sumitayadav9407 3 роки тому

    The best YARN explanation . Thank you !

  • @ajitsharma8991
    @ajitsharma8991 3 роки тому

    Best YARN Lecture. Sir, Can you please explain? How to configure capacity-scheduler for queue for submit multiple spark application at same time? Or any other suggestion to run multiple application on Hadoop yarn cluster. Please

  • @piyushsharma1040
    @piyushsharma1040 4 роки тому

    The best YARN lecture ever! 45 minutes are completely worth it!

  • @SrikantShirisha
    @SrikantShirisha 4 роки тому

    Another question I have is what is the resource manager died but the data was processed and manipulated and is no longer in its original state?

  • @SrikantShirisha
    @SrikantShirisha 4 роки тому

    Difference between yarn and map reduce?

    • @abusufiyan930
      @abusufiyan930 3 роки тому

      map reduce is single point of failure yarn is not single point

  • @pranalinikumbh6185
    @pranalinikumbh6185 4 роки тому

    superb explaination

  • @junaidansari675
    @junaidansari675 4 роки тому

    Great explanation, make videos for other services, hadoop admin etc, issues as well

  • @alihammadshah
    @alihammadshah 4 роки тому

    Oulryte - goodbye

  • @srikrishnarr6553
    @srikrishnarr6553 4 роки тому

    Thanks professor ..Your throat deserves some rest yes !!Nice explanations

  • @bvb6914
    @bvb6914 4 роки тому

    How many container could be launch in one node?

  • @tapobanjit
    @tapobanjit 4 роки тому

    First of all your fsimage is not present on the shared storage (i.e Journal Nodes), it only contains edit logs which records changes in HDFS fs. Both the active as well as standby NNs store the fsimage on the local disk hosting the daemons.

  • @bobbyvenkatesan3657
    @bobbyvenkatesan3657 4 роки тому

    Awesouy. Right video to learn Yarn

  • @tushibhaque863
    @tushibhaque863 4 роки тому

    Awesome

  • @tushibhaque863
    @tushibhaque863 4 роки тому

    Thanks for sharing your knowledge

  • @bobbyvenkatesan3657
    @bobbyvenkatesan3657 4 роки тому

    Great video. Thank you so much

  • @ganeshgokhale1657
    @ganeshgokhale1657 4 роки тому

    please try to explained Hadoop Resource manager High availability in deep.

  • @yasim9435
    @yasim9435 4 роки тому

    When ApM no requested container at the node where data is stored, but RM not able to find any , ApM is given container and has to process data over remote access. In case of network failures in remote access, who and what actions are taken?

  • @avigarg5525
    @avigarg5525 5 років тому

    one of the best I have seen for yarn understanding..(Y)

  • @ishanvarshney9054
    @ishanvarshney9054 5 років тому

    much better than edureka!

  • @nareshbabu9235
    @nareshbabu9235 5 років тому

    Very nice

  • @dheerajkumarsolanki6220
    @dheerajkumarsolanki6220 5 років тому

    In 1st scenario, if 2nd block is present in different node. Then how the 3rd line is read by the record reader?

  • @nareshbabu9235
    @nareshbabu9235 5 років тому

    Very very good Explanation

    • @Selfreflex
      @Selfreflex 5 років тому

      Thanks

    • @nareshbabu9235
      @nareshbabu9235 5 років тому

      @@Selfreflex Can you please upload videos on Spark 2

  • @nabilraza456
    @nabilraza456 5 років тому

    False !!

  • @jayanthkumar4605
    @jayanthkumar4605 5 років тому

    Crystal Clear explanation, in Simple English . Thumbs up to you. Great tutorial for HDFS. I understood a lot from this video. Thank you so much.

  • @keeranmnc1605
    @keeranmnc1605 5 років тому

    @44:13 - Kumar - can't stop laughing

  • @gauravpratap4482
    @gauravpratap4482 5 років тому

    how the recordreader will know the starting of data if the data is going in block2?

  • @arghamaz
    @arghamaz 5 років тому

    Excellent explanation...simple