AWS Glue, DataBrew, Lambda and Athena වැඩ්ඩන්ගේ උදව් ඕනි

[lorensuhewa]

Machan.....
I'm new to AWS and I got this assignment from our company to do.

I managed to do part of this process by myself using google help and YouTube materials. Still, I have few unclear arears where I need to clarify. Is there anyone who can help me please revert.

 

[Nidarshana_k]

@හෙළයෙක් @HAneo @kasun_kbp

 

[Dr.Sheldon_Suprakata]

up

 

[gulp]

Post exactly you are unclear of here so in future someone searching the same can also benefit

 

[Nidarshana_k]

bump

 

[undefined]

bump

 

[windows_ubuntu]

Bump

 

[Nadunf]

Bump

 

[kkroxc]

Nikan Karala dennada kiyanne?

 

[lorensuhewa]

Nikan Karala dennada kiyanne?

Podi unclear than tikak thiyenawa..... ekata support ekak oni....

---

Post exactly you are unclear of here so in future someone searching the same can also benefit

will post a detailed on again.
------ Post added on Jul 15, 2023 at 7:06 PM

 

[visalbro]

Bump

 

[හෙළයෙක්]

unclear තැන් ටික දාන්න මචන්.

 

[kaizoku_O]

bump

 

[lorensuhewa]

unclear තැන් ටික දාන්න මචන්.

machan mekai case eka.....

Lambda function eka every 5 mins run wenawa.
ethakota API eken latest seven days wala data each run ekakadi kiyawala S3 bucket ekata daagannawa.

Prashne thiyenne machan....kohomada No.5 process ekedi duplicates ayin karanne kiyala...
palaweni data set eke duplicates ayin karala apahu S3 eke save karala. ilanga data set eke apahu duplicate thiyenawane.

Glue eke koi weleth check karanne eka data set ekaka duplicates witharai.....

THerunada machan case eka?

 

[හෙළයෙක්]

machan mekai case eka.....

Lambda function eka every 5 mins run wenawa.
ethakota API eken latest seven days wala data each run ekakadi kiyawala S3 bucket ekata daagannawa.

Prashne thiyenne machan....kohomada No.5 process ekedi duplicates ayin karanne kiyala...
palaweni data set eke duplicates ayin karala apahu S3 eke save karala. ilanga data set eke apahu duplicate thiyenawane.

Glue eke koi weleth check karanne eka data set ekaka duplicates witharai.....

THerunada machan case eka?

P1 එකේ Data set එක dups අයින් කරල Save කරනව D1
P2 එකේ Data Set eka dups අයින් කරල save කරනව D2

දැන් ඔයාට ඕනි D2 save කරන්න කලින් dups චෙක් කරද්දි D1 එකත් එක්කම චෙක් කරන්නද?

 

[ThisaraMalintha]

bump

 

[TNHM]

data duplicate kiyala hoyaganne kohomada. PK ekak thiyenawada

 

[NRTG]

BUMPI

 

[Djice]

machan mekai case eka.....

Lambda function eka every 5 mins run wenawa.
ethakota API eken latest seven days wala data each run ekakadi kiyawala S3 bucket ekata daagannawa.

Prashne thiyenne machan....kohomada No.5 process ekedi duplicates ayin karanne kiyala...
palaweni data set eke duplicates ayin karala apahu S3 eke save karala. ilanga data set eke apahu duplicate thiyenawane.

Glue eke koi weleth check karanne eka data set ekaka duplicates witharai.....

THerunada machan case eka?

5 min schedule ekaka run wena lambda ekakin ai oi past 7 days wala data fetch karanne? wtf

glue wala duplicate data din karanda feature ekak thiyanawa - https://aws.amazon.com/glue/features/

 

[lorensuhewa]

P1 එකේ Data set එක dups අයින් කරල Save කරනව D1
P2 එකේ Data Set eka dups අයින් කරල save කරනව D2

දැන් ඔයාට ඕනි D2 save කරන්න කලින් dups චෙක් කරද්දි D1 එකත් එක්කම චෙක් කරන්නද?

hariyata hari machan. meka thama case eka.

D2 set eka wena Glue job ekak daala D1 ekath ekka check karannath puluwan. but machan meka loop ekak wage grow wenawane digatama. man dan karanne D2 apahu D1 ekath ekka check karala DD kiyala save karanawa. DD kiyana eka hamawelama replace karanawa parana data set ekata.

habai data grow weddi meka reliable solution ekak nemei kiyalai mata hithenne. mokada DD eka winadi 5n 5ta grow wena hinda data wadi unama write karanna yana processing time eka wadi wena eka saha prosseing time eka over 5 mins giyoth ilanga run eka wenne nathi eka.

---

5 min schedule ekaka run wena lambda ekakin ai oi past 7 days wala data fetch karanne? wtf

glue wala duplicate data din karanda feature ekak thiyanawa - https://aws.amazon.com/glue/features/

meka thama machan man danata use karanne. but data set eka grow wena hinda glue feture eka hariyatama use karanna baha. mokada Glue eka dups check karanne dataset wise.

past 7 days wala data thamai client koi weleth ewanne. Lambda ekata puluwan nam kalin run una latestma data set record eka API call ekata pass karala aluth data tika witharak illanna...ehenan wade hari. but mata therenne naha ehema karanna puluwanda barida kiyala....
------ Post added on Jul 17, 2023 at 12:48 PM

---

data duplicate kiyala hoyaganne kohomada. PK ekak thiyenawada

Glue ekei DataBrew ekei meka karanna puluwan machan.
------ Post added on Jul 17, 2023 at 12:49 PM