Видео

nice video! what's the point of using jobs in notebooks since bookmarks aren't supported there? is there another benefit?

(a bit late to this video) But still - okay, very good - well done. I now wish I had come across you a year ago - I see from your videos, you do a good job. NO massive amounts of annoying face time in the videos as too many others do. Clear and concise info in the videos AND great narrating voice. I went through a lot of pain to first, obtain the GCP Pro Data Engineer cert last year - but because of NOT having the REQUIRED "multiple years of hands-on GCP data engineering work experience" companies want AND not many GCP DE roles near me (not doing full remote yet) AND the fall tech layoffs - I had to rotate back to AWS world (after a decade as an AWS Cloud Architect/Engineer - renewed Architect Pro cert last year). Spent the past 5 months studying my hinie off and then gaining the AWS DE cert (the reason why I wrote my June 15th paper - tinyurl.com/48e6skab to help others out, to 'maybe' expedite their path...). Now the problematic deal is to find a company who will take me on with the double DE certs and without the REQUIRED "multiple years of hands-on data engineering work experience". My thoughts have been, I should be a magnet to multiple companies after gaining these two supposedly difficult DE certs... AWS was difficult and did not seem to be an Associate level cert (I've acquired other AWS Pro certs and this AWS Data Engineer cert most definitely seemed at that level - it pissed me off because I failed the exam back in April...).

as I am working on glue, it super argent to me, the question is, how to add the source file name in the dyamaic frame, it would be great if you could respond on a priority basis. Thank you

how to get input file name of a record in spark dynamic dataframe ?

Amazing Man!! Good one

thank you for making videos like this.

Thanks, sir !!!!!

wondering if watching costs are really a data engineers activity?

Still a great overview. Makes everything a lot clearer. Thank you.

This is fantastic! I'm new to AWS Glue and was really struggling to get traction developing an ETL script. Being able to develop locally, I don't really care about the costs, but the ability to debug, get feedback, and just the turnaround time to try things is amazing. Again, thanks. I'd like to ask you more questions, how can I do that?

i guess you forgot to show how to make a connection in pgadmin first

Now AWS includes AWSSDKPandas-Python312 so it's easier to add pandas to your Lambda function. However, I'm getting "Missing optional dependency 'fsspec'. Use pip or conda to install fsspec." I've follow these steps to install the latest version, but failed: "Unable to import module 'lambda_function': No module named 'fsspec'". Any suggestions?

The concepts in this video went inside my brain like a hot knife going in butter. Great video for someone like me who comes from a functional background. Great work...really appreciated.

I am still facing with the numpy import error and even used AWSLambda-Python38-SciPy1x library , but unable to resolve , kindly let me know if any inputs

i appreciate the really informative video! I followed everything and i'm stuck on running the pyspark code in the dev container environment. when i launch the dev container in the same/new window, i don't see the extensions in the container environment. The python interpreter doesn't show up either and when i go to the extension tab in the container environment, all the extensions are not installed. Is there additional configuration files in vs code i need to modify to enable the already installed extensions to run from the dev container?

I'm a newcomer to AWS a data engineer, Your channels really help me! Appreciate it!

love this

You're an excellent teacher, keep it up...subscribing now

Thanks for the video! I've tested the entire workflow, but I'm encountering an issue with the section on creating a DynamicFrame from the target Redshift table in the AWS Glue Data Catalog and displaying its schema. While I can see the updated schema reflected in the Glue catalog table, the code you provided still prints the old schema.

thank you man!!!

Can you please make a video showing practical implementation of event driven architecture using event bridge, sns, sqs and lambda? Thank you!

one of the best channels on youtube for data engineering! great content, keep the great work

Can you please make a video showing implementation of event driven architecture using event bridge, sns, sqs and lambda? Thank you!

how do i get to personalize the message that sns sends?

can lakeformation be created from IaC? like cloudformation/terraform?

Thank u so much. Your tutorial helps me a lot.

Data bricks glue are same?

I able to run successfully with glueContext.write_dynamic_frame.from_options & glueContext.write_dynamic_frame.from_jdbc_conf but I see some issue with glueContext.write_dynamic_frame.from_catalog as below: Getting below error: Error Category: UNCLASSIFIED_ERROR; An error occurred while calling o130.pyWriteDynamicFrame. Exception thrown in awaitResult: SQLException thrown while running COPY query; will attempt to retrieve more information by querying the STL_LOAD_ERRORS table Could you please guide

Thank you for this guide! I tried using pycharm and my old computer cannot handle it. I almost purchased for a subscription.

Beautifully explained about the setup. Understood how docker works as well. thanks a ton, Subscribed.

Docker option not available in PyCharm community edition I guess

Didnt understand why cost incured? As its running locally & why to keeep nodes set to 2

Very much helpful. Thanks

Is possible to do the same thing but with lines features?

Hello. I made a Glue Job that performs ETL changes to data in an S3 Bucket and exports the changed data to a Redshift database, but now I'm thinking of changing from Redshift to PostgreSQL. I know this video is for importing RDS data into Glue, but if I follow the video's instructions, would I also be able to export it back into RDS?

Thank you!!

thank you for the video. I am unable to run the program because i do not see the run button after clicking "attach in current window"

Thanks for this tutorial! I have a question: How do I store one big parquet file in s3 without running with Kernel dying issues? I have already use df.convert_dtypes() but it is still failing. My file has over 1.5 million rows. Files below 900k rows are not failing! Thanks

sorry, what is wrong with df = spark.read.csv(path)?

Clear and concise

Just found your channel. can we have a complete playlist , a type of course or a oneshot video/videos, your explain in depth and I found your videos better than the other tutorials on youtube

How would you recommend local dev and organization -> uploading to AWS for scripts with multiple files ?

Thank you for this very useful video!

how to use python in FME?

Your functions based job is quite straightforward. Would you like to organize your glue job in a Python class?

Do you do any consulting?

how do you add third party python libraries ?

does this work with the debugger?

Thank you for the great work. I am new to Glue and your videos are great help. I was able to create derived column based on this video like new column - total_amount which is equal to price * quantity. Now in next step I want to categorize the customers based on total_amount i.e. if total_amount <300 then "small" else if total_amount >=300 and total_amount<500 then "Medium" else "large. I have defined function for this by passing dynamic dataframe but when i am executing that I am just getting root. ALso, if I run the whole script it is completing without any error but also not giving desired result. Could you please confim if we can use this conditional logic to get the derived column. Thank you in advance.

Do you use the Factory Design pattern?