- Indima yoNjiniyela Wedatha igxile ekuklameni nasekulondolozeni izinhlelo eziqoqa, ziguqule, futhi zigcine idatha ngendlela ethembekile nengalinganiswa.
- Indlela yokufunda ihlelwe ngamazinga: izinhlelo kanye nezizindalwazi, i-Big Data kanye namapayipi, bese ekugcineni ifu, ukuphepha kanye nokusakaza.
- Ukwazi i-SQL, ukumodela idatha, i-ETL, ukuhlelwa kwemisebenzi, izitsha, kanye nomhlinzeki wefu okungenani oyedwa kuyisihluthulelo sentuthuko yobungcweti.
- Amaphrojekthi asebenzayo, izindawo zokugcina ulwazi emphakathini, kanye nezitifiketi kusiza ukuhlanganisa ulwazi nokuthuthukisa izinketho zokufuna umsebenzi.
Indlela yomsebenzi wokuba uNjiniyela Wedatha isibe ngenye yezinto ezikhanga kakhulu emhlabeni wedatha, ikakhulukazi kulabo abavela ezizindeni ezifana ne-Data Analyst noma i-Data Scientist abafuna ushintsho lobuchwepheshe. Izinkampani eziningi zidinga abantu abakwazi ukuklama, ukwakha, nokugcina izinhlelo eziqhuba ulwazi, hhayi nje amamodeli okufunda komshini noma amadeshibhodi.
Ngesikhathi esifanayo, inani elikhulu lezinsizakusebenza, izifundo, kanye nezincomo ezisakazwa ku-inthanethi zingaba nzima kakhulu: ukuthi uqale nge-Python, ufunde i-SQL kanye nokubonisa kuqala, uye ngqo efwini, noma ushintshele ku-Spark… Kulesi sihloko, uzothola indlela yokufunda ephelele ngesiSpanishi, esekelwe ezintweni zokubhekisela futhi yandiswa ngomongo osebenzayo, ukuze wazi kahle ukuthi uqale kuphi, ukuthi ungathuthuka kanjani, nokuthi yiziphi izinqumo okufanele uzenze ekuthuthukisweni kwakho njengoNjiniyela Wedatha.
Uyini uNjiniyela Wedatha futhi kungani indima yakhe ichuma?
Unjiniyela Wedatha unesibopho sokuklama, ukwakha, nokusebenzisa izinhlelo eziqoqa, eziguqula, ezigcina, futhi zenze kutholakale idatha esetshenziswa yizinkampani ukwenza izinqumo. Ngenkathi usosayensi wedatha egxila kakhulu kumamodeli nasekuhlaziyweni, unjiniyela wedatha uqinisekisa ukuthi ulwazi lulethwa ngesikhathi, ngokuthembekile, ngokukhula, nangokuphephile.
Empeleni, umsebenzi wansuku zonke woNjiniyela Wedatha uvame ukuhlanganisa ukwakha amapayipi e-ETL noma e-ELT, ukuhlela izinqubo, ukuklama izakhiwo zedatha (amachibi edatha, izindawo zokugcina idatha, ama-datamart) , ukuhlanganisa imithombo eminingi, kanye nokusebenzisana namanye amaqembu afana nokuhlaziya, isayensi yedatha, noma umkhiqizo.
Ngokusho kwemibiko ehlukahlukene yemboni, isidingo sonjiniyela beDatha siyaqhubeka nokukhula , futhi imiholo yabo ngokuvamile iphakeme kuneyochwepheshe besayensi yedatha ezimakethe eziningi, ngenxa yomthelela oqondile abanawo engqalasizinda yezobuchwepheshe kanye nekhono lenkampani lokusebenzisa idatha yayo.
Amapulatifomu agxile ekuqeqeshweni kwedatha aqokomisa ukuthi imisebenzi engaphezu kuka-70% yonjiniyela wedatha idinga ulwazi oluqinile lobunjiniyela besofthiwe nezinhlelo ezisatshalaliswayo , nokuthi amazinga omholo walo msebenzi angadlula kalula lawo amanye amaphrofayili okuhlaziya lapho kuhlanganiswa amakhono okuhlela, amafu, kanye nokwakha.
Kusukela kuSosayensi Wedatha kuya kuNjiniyela Wedatha: kungani abaningi beshintsha
Ezinhlanganweni eziningi, ikakhulukazi izinkampani ezintsha noma izinkampani ezikhulayo, imigqa phakathi kwe-Data Scientist kanye ne-Data Engineer ayicacile. Ngokuvamile, amamodeli okuqeqesha umuntu kufanele ahlanze idatha, akhe izikripthi zokukhipha, ahambise amafayela, enze izinqubo ngokuzenzakalela, futhi amise ngisho nama-API ukuze alethe izibikezelo.
Uma uke wazithola wakha amapayipi, usebenzisa amamodeli ngesandla, noma uxhumanisa imithombo eminingi yedatha , kungenzeka ukuthi usuvele usebenza eduze kakhulu nalokho okwenziwa yi-Data Engineer. Lokhu kuvezwa kobuchwepheshe kuvame ukuvusa isithakazelo sokuqonda yonke indlela yokusebenza, kusukela ekungenisweni kwedatha kuya ekukhiqizweni, kanye nokunciphisa ukuthembela kwamanye amaqembu noma izixazululo zesikhashana.
Isizathu esiyinhloko salolu shintsho ukuzimela kobuchwepheshe : uma uqonda ukuthi amapulatifomu edatha aklanywe kanjani, ukuthi yibuphi ubuchwepheshe obukhona, nokuthi asetshenziswa kanjani efwini, ungaletha imibono yakho ekukhiqizweni ngokuqinile, ngaphandle kokubhajwa ezincwadini zokuhlola ezingafiki kumsebenzisi wokugcina.
Ngaphezu kwalokho, imakethe yemisebenzi ifuna kakhulu amaphrofayili obunjiniyela bedatha . Nakuba izikhundla zesayensi yedatha kuphela zivame ukuzinza, isidingo sabantu abakha ingqalasizinda yedatha, amapayipi esikhathi sangempela, kanye nezinhlelo ezinwebekayo sikhula njalo, okwenza ushintsho lube yisinqumo esihlakaniphile seminyaka ezayo.
Amazinga omzila wobungcweti: abaqalayo, abaphakathi, nabathuthukile
Ukuze ugweme ukukhungathekiswa ulwazi oluningi kangaka, kuyasiza ukuhlukanisa indlela ye-Data Engineer ibe amazinga amathathu okuvuthwa: abaqalayo, abaphakathi, kanye nabathuthukile. Umqondo awukona ukukukhohlisa, kodwa ukukusiza ukuthi ubheke phambili ukuthi yini okufanele uyifunde kuqala ngokusekelwe endaweni yakho yokuqala.
Izinga labaqalayo lihlanganisa izisekelo: ukuhlela, ukuqonda, ukulawula inguqulo, kanye nezizindalwazi eziyisisekelo. Yilokhu okudingayo uma uqala kusukela ekuqaleni noma uvela esizinda esingenabo ubuchwepheshe obutheni, njengendima yebhizinisi noma yomhlaziyi.
Izinga eliphakathi lihlanganisa izihloko ezifana ne-Big Data, amathuluzi okucubungula asakazwayo, ukwakheka kwamapayipi e-ETL, kanye nabahleli. Lapha uqala ukuhlola ubuchwepheshe ozobubona ezindaweni zokukhiqiza nokucabanga njengomklami wedatha.
Izinga elithuthukile lifaka amakhono e-cloud computing, izitifiketi, ukuphepha, ukuthunyelwa okuqhubekayo, ukusakaza bukhoma ngesikhathi sangempela, kanye nokufuna umsebenzi kanye nokulungiselela izingxoxo zobuchwepheshe . Lesi yisigaba lapho uhlose khona izikhundla eziphezulu noma ezikhethekile.
Njengomthetho ojwayelekile, uma ungakabi nekhono ekuhleleni izinhlelo , kunengqondo ukuqala ngesigaba se-Programming and Databases. Uma usuvele ukhululekile nge-SQL kanye ne-Python ethile, ungadlulela ngokushesha ku-Big Data and Data Processing. Futhi uma umgomo wakho uwukuqinisekiswa kwefu, isigaba sefu sizoba sibalulekile.
Izisekelo zokuhlela kanye nokulawulwa kwenguqulo
Isisekelo cishe sayo yonke into kubunjiniyela bedatha ukwazi ukuthi ungahlela kanjani ngempumelelo . Akukhona nje ngokubhala izikripthi "ezisebenzayo," kodwa mayelana nokudala ikhodi elungisekayo, efundekayo, futhi elula ukuyilungisa. Kule ndawo, i-Python ivame ukuba yindawo engcono kakhulu yokungena ngenxa ye-syntax yayo elula kanye ne-ecosystem yayo enkulu kwisayensi yedatha kanye nobunjiniyela bedatha.
Kulesi sigaba, kubalulekile ukuhlolisisa imiqondo eyisisekelo yokuhlela : izinhlobo zedatha, izakhiwo (uhlu, izichazamazwi, amasethi), imisebenzi, amakilasi, ukuphathwa kwamaphutha, kanye nokufunda nokubhala amafayela. Uma ukhetha ezinye izilimi ezifana neJava, iScala, i-R, noma iJulia, lezo nazo ziyasebenza, kodwa ezweni langempela lobunjiniyela bedatha, iPython neJava/Scala yizona ezihamba phambili.
Ngesikhathi esifanayo, ukufunda ukulawula inguqulo nge-Git kubalulekile . Abaningi bakubona njengokuwusizo kuphela ekusebenzeni ngokubambisana, kodwa empeleni kukuvumela ukuthi ulandelele umlando wekhodi yakho, uqonde ukuthi yini eshintshile nokuthi nini, uvivinye imibono ngaphandle kokwesaba, futhi ugcine umsebenzi wakho uhlelekile. I-GitHub noma i-GitLab izoba yizinkundla zakho zansuku zonke zokubamba izindawo zokugcina nokubambisana.
Akudingeki ube yi-Git guru kusukela ngosuku lokuqala, kodwa udinga ukwazi kahle imiyalo eyisisekelo (init, add, commit, branch, merge, push, pull) futhi uqonde ukuthi amagatsha, izicelo zokudonsa, kanye nokubuyekezwa kwekhodi kusebenza kanjani. Le ndlela yokusebenza iyinto evamile kunoma yiliphi ithimba lobuchwepheshe elingathi sína.
Ama-database, i-SQL, kanye nokumodela kolwazi
Uma ususungule isisekelo esiqinile ekuhleleni izinhlelo, sekuyisikhathi sokubhekabheka kudathabheyisi kanye ne-SQL . Yilapho abantu abaningi bedideka khona mayelana nohlelo: i-Python kuqala, bese kuba yi-SQL, noma okuphambene nalokho? Indlela enengqondo kakhulu ukuthuthuka ngokuhambisana, kodwa ukuqinisekisa ukuthi ukusebenzisa i-SQL kuba yinto yesibili kuwe.
Ngedatha ehlelekile, i-PostgreSQL iyindawo yokuqala enconywa kakhulu ngenxa yamandla ayo kanye nesimo sayo njengendinganiso yangempela kumaphrojekthi amaningi. Uma usuvele ujwayelene ne-MySQL, i-SQLite, noma ezinye izinjini zedatha, zisazosebenza, yize i-PostgreSQL ivame ukunikeza ukuguquguquka okwengeziwe ezindaweni zobungcweti.
Kungumqondo omuhle futhi ukuzijwayeza ngemininingwane ye-NoSQL , njenge-MongoDB yamadokhumenti noma i-Redis yamabhangqa e-key-value, kanye nezinye ezifana ne-Cassandra yamakholomu. Umgomo akukhona ukuwakhumbula wonke, kodwa ukuqonda izimo zokusetshenziswa kwawo, izinzuzo, kanye nokungalungi, nokwazi ukuthi kufanele ukhethe nini enye kunenye.
Lesi sigaba sethula ukumodela kwedatha : imodeli yobudlelwano, imodeli yobukhulu, imiqondo yamaqiniso nobukhulu, ukujwayelekile, okhiye abayinhloko nabangaphandle, kanye nobuqotho bokubhekisela. Uzofunda ukucabanga ngokwezinhlelo zethebula, ubudlelwano, kanye nemibuzo ephumelelayo, okubalulekile kunoma yiluphi uhlaka lwesakhiwo olulandelayo.
Kamuva, uzongena ujule emachibini edatha, ezindaweni zokugcina idatha, kuma-data marts, nasezindaweni zedatha , kanye nezindlela ezifana nesitoreji sekholomu ngokumelene nomugqa, ama-star schemas, ama-snowflake schemas, kanye namasu e-read vs write schema. Konke lokhu kuzokunikeza ulimi namaphethini asetshenziswa kumaphrojekthi wangempela ukuhlela ulwazi ngezinga.
Imiqondo ye-Big Data, i-analytics kanye ne-business intelligence
Njengoba uqonda kahle izisekelo ze-SQL kanye nedathabheyisi, kungumqondo omuhle ukuhlola imiqondo ye-Big Data kanye nokuhlaziya . Akudingeki ube uchwepheshe kuzo zonke izinhlaka ohlelweni lwe-ecosystem, kodwa kufanele uqonde ukuthi yiziphi izinkinga abahlose ukuzixazulula nokuthi kungani zikhona.
Izwe le-Big Data lithembele ekucutshungulweni okusatshalalisweni , lapho esikhundleni sokusebenzisa yonke into emshinini owodwa, umsebenzi usatshalaliswa kuma-node amaningi. Amathuluzi afana ne-Apache Spark asethandwa kakhulu ekucubunguleni inani elikhulu ledatha, kokubili ku-batch kanye nasekusakazeni, futhi avame ukuba yingxenye yezinqwaba zobuchwepheshe zezinkampani eziqhutshwa idatha.
Ngaphezu kwe-Big Data, kuyazuzisa ukuthola ukuqonda okujwayelekile kobuhlakani bokwenziwa, ukufunda komshini, kanye nobuhlakani bebhizinisi . Ngenkathi njengoNjiniyela Wedatha ungeke uqeqeshe amamodeli ayinkimbinkimbi, uzoba nesibopho sokulungiselela idatha yawo nokuklama ingqalasizinda ewondlayo.
Uzobona nokuthi izinto ezifana namathuluzi e-BI (i-Power BI, i-Tableau, i-Looker, njll.), izinqubo zokubika, kanye nezidingo zabahlaziyi bebhizinisi zihlangana kanjani. Ukuqonda imisebenzi yabo kuzokusiza ukuthi uklame amapayipi edatha awusizo kakhulu kanye namamodeli kulabo abasebenzisa ulwazi.
Ukucutshungulwa kwedatha: I-ETL, ukuhlelwa kanye nemibhobho yedatha
Inhliziyo yangempela yobunjiniyela bedatha ukuklama nokwakhiwa kwamapayipi edatha . Lapha uzofunda kahle ukuthi iyini ipayipi le-ETL (Extract, Transform, Load), lapho indlela ye-ELT inengqondo, ukuthi ungayihlela kanjani imisebenzi, uyiqaphe, futhi ululame ekuhlulekeni.
Ipayipi elijwayelekile lihlanganisa izigaba zokungenisa idatha emithonjeni eminingi (ama-API, izizindalwazi, amafayela, imigqa yemiyalezo), izinyathelo zokuhlanza nokuguqula (ukwenza kube ngokwejwayelekile, ukuhlanganiswa, ukucebisa) bese ekugcineni kulayishwa ohlelweni oluthile oluqondiwe, okungaba yindawo yokugcina idatha, ichibi ledatha, izizindalwazi ze-NoSQL noma ingxube yezinhlobo eziningana.
Kulesi simo, amathuluzi okuqondisa ukuhamba komsebenzi afana ne-Apache Airflow nezinye izindlela zesimanje avele, okuvumela abasebenzisi ukuthi bachaze ukuncika phakathi kwemisebenzi, ukuhlela ukwenziwa, ukulandelela ukuthi yini eyenziwe, kanye nokusabela emaphutheni. Nakuba inkampani ngayinye isebenzisa inqwaba ehlukile, indlela yokucabanga yokuhlela nokwenza izinqubo ngokuzenzakalela ivamile kubo bonke.
Iphuzu elibalulekile ukuqonda ikhathalogi yemiqondo evame ukusetshenziswa kulezi zindawo: amamodeli ahlobene nawobukhulu, amachibi edatha, ama-data marts, izindawo zokugcina idatha, ukwakheka kwamakholomu nomugqa, izikimu zezinkanyezi nezeqhwa , kanye namasu okufunda nokubhala ama-schema ahlukene. Ukuqonda kahle leli gama kuzokuvumela ukuthi uqonde imibhalo yobuchwepheshe, izincwadi ezikhethekile, kanye nezakhiwo ozibonayo kumadayagramu.
Lesi sigaba singenye yalezo ezizuza kakhulu ekuzilolongeni okusebenzayo kanye namaphrojekthi amancane omuntu siqu, lapho ungakha khona amapayipi asuka ekugcineni aye ekugcineni , noma ngabe usebenzisa idatha yomphakathi, futhi uzijwayeze amaphethini ajwayelekile ozowabona kamuva ezindimeni zobungcweti.
Ukuphepha emipayipini yedatha kanye namapulatifomu
Isinyathelo sokuqala ukusebenzisa isimiso selungelo elincane kakhulu ezindimeni nasezimvumeni : i-akhawunti ngayinye yesevisi, umsebenzisi, noma uhlelo lokusebenza kufanele lube nokufinyelela okudingekayo kuphela ukuze lwenze umsebenzi walo, hhayi okunye. Lokhu kunciphisa indawo yokuhlasela futhi kunciphise umthelela wamaphutha noma ukuvuza.
Kubalulekile futhi ukuqonda ukuthi ukubethela idatha kusebenza kanjani ekuthuthweni nasekuphumuleni . Lokhu kufaka phakathi ukusebenzisa i-HTTPS, i-TLS, kanye nezinye izinqubo ezivikelekile lapho uhambisa idatha phakathi kwezinsizakalo, kanye nokuvumela ukubethela kudathabheyisi, amabhakede okugcina, kanye nezinye izinhlelo lapho kugcinwa khona ulwazi.
Uma udalula ama-API noma izinsizakalo zamamodeli, kufanele unake imininingwane efana nokuqinisekisa kanye nokugunyazwa (amathokheni, okhiye be-API, i-OAuth, njll.), ukhawulele ukufinyelela kuma-endpoints abalulekile, kanye nomsebenzi wesistimu yelogi ukuze uhlole ukusetshenziswa kabi. Awudingi ukuba uchwepheshe wezokuphepha, kodwa udinga izinga elanele lobuchwepheshe ukuze wenze izinqumo ezinomthwalo wemfanelo.
Konke lokhu akugcini nje ngokuvimbela izimanga, kodwa futhi kuqinisa iphrofayili yakho yobungcweti emehlweni enkampani, njengoba ubonisa ukuqaphela umthelela wangempela womsebenzi wakho ebhizinisini kanye nokuvikelwa kwedatha yamakhasimende kanye neyomsebenzisi.
Izinhlobo zomklamo wesakhiwo sokugcina kanye nedatha
Uma ushintsha kusuka ekusebenzeni ngamasethi edatha angaguquki njengososayensi wedatha uye ekubeni unjiniyela wedatha, ubudlelwano bakho nesitoreji bushintsha ngokuphelele . Akusekho mayelana nokuvula ifayela le-CSV endaweni, kodwa mayelana nokuklama izinhlelo ezisekela ukugeleza kwedatha okuqhubekayo, ama-schema aguqukayo, kanye nabathengi abaningi ngasikhathi sinye.
Emsebenzini wakho wansuku zonke uzohlanganisa izinhlobo ezahlukene zesitoreji: izizindalwazi ezihlobene (i-PostgreSQL, i-MySQL) zolwazi oluhlelekile nolwentengiselwano; izizindalwazi ze-NoSQL ezifana ne-MongoDB (amadokhumenti), i-Redis (inani eliyisihluthulelo) noma i-Cassandra (amakholomu) ngezidingo ezithile zokusebenza, ukuguquguquka kweskimu noma ukukala okuvundlile.
Ngaphezu kwalokho, isitoreji sezinto esisekelwe efwini (i-Amazon S3, i-Azure Data Lake Storage, i-Google Cloud Storage) sesibe yitshe lesisekelo lamachibi amaningi edatha anamuhla. Inqwaba yedatha eluhlaza necutshungulwayo igcinwa lapha, ngokuvamile ngamafomethi afana neParquet noma i-Avro, alungele ukusetshenziswa yizinjini ezahlukene zokuhlaziya.
Ukuklama ukwakheka kwedatha yesimanje kuhilela ukucabangela indlela idatha egeleza ngayo isuka emithonjeni iye kubathengi, ukuthi yiziphi izendlalelo eziphakathi zekhwalithi, ukubusa, noma ukuguqulwa okudingekayo, nokuthi ungahlela kanjani konke lokhu ukuqinisekisa ukuthi kuyagcinwa. Ukukwazi ukufunda nokudala imidwebo yokwakha kuzoba yingxenye evamile yomsebenzi wakho.
Ngaphezu kwalokho, izinhlangano eziningi zamukela izakhiwo ezigxile ekusakazeni, lapho ubuchwepheshe obufana ne -Apache Kafka budlala indima ehamba phambili njengomgogodla wemicimbi, okusiletha esigabeni esilandelayo.
Ukusakaza kanye nokucubungula ngesikhathi sangempela nge-Apache Kafka
Ukuhlaziywa kwedatha okujwayelekile okuningi kwenziwe ngemodi ye-batch: ukulayisha idatha njalo, ukuyicubungula, nokukhiqiza imiphumela . Kodwa-ke, izinkampani eziningi zidinga ukusabela ngesikhathi sangempela kulokho okwenzekayo, kusukela ekuthengiselaneni kwezezimali kuya emsebenzini womsebenzisi noma izinzwa ze-IoT.
Kulesi simo, i-Apache Kafka yavela njengeplatifomu yokusakaza imicimbi eyamukelwa izinhlangano eziyizinkulungwane ezingamashumi emhlabeni jikelele. I-Kafka ivumela abasebenzisi ukuthi bashicilele futhi basebenzise imiyalezo ezihlokweni, ngabakhiqizi nabathengi abahlukanisiwe, futhi bakhulise uhlelo ukuze luphathe kusukela kwezigidi ezimbalwa kuya kwezigidi zemicimbi ngomzuzwana.
KuNjiniyela Wedatha, ukuqonda okuqinile ngokwakhiwa kweKafka kubalulekile: ukuthi yiziphi izihloko, izingxenye, abadayisi, abakhiqizi, abathengi, amaqembu okusebenzisa, kanye nezindlela zokunciphisa. Kubalulekile futhi ukuqonda ukuthi ungahlanganisa kanjani iKafka nezinhlelo ezingezansi (izizindalwazi, izindawo zokugcina idatha, izinhlelo zokuxwayisa) kanye nezinqubo zokuhlaziya zesikhathi sangempela.
Amamodeli amaningi okufunda komshini manje asebenza emifudlaneni yedatha, okudinga ukuhlanganiswa kwama -MLOp namapulatifomu okusakaza ukuze kunikezwe izibikezelo ezibukhoma. I-Kafka ayiseyona nje "eyobunye ubuchwepheshe" kodwa isiba yisisekelo sezakhiwo zesimanje eziqhutshwa yimicimbi.
Abaphathi be-IT ezinkampanini ezinkulu babheka izinhlelo zokusakaza njengesici esibalulekile samasu abo edatha kanye ne-AI , babika ukuthuthuka okukhulu ekubuyiselweni kokutshalwa kwezimali lapho besebenzisa lezi zakhiwo. Ukufunda i-Kafka kanye nemibono ehlobene kukubeka phambili kunabanye abantu abaningi abazongenela ukhetho.
Ama-Container, i-Docker, kanye nokuthunyelwa kwesevisi
Ekushintsheni kusuka kusosayensi wedatha ukuya kunjiniyela wedatha, iphuzu elibalulekile ukuqonda kahle ukupakishwa nokusetshenziswa kwezinsizakalo nge-Docker . Uqala ukusebenzisa izikripthi emshinini wakho uye ekwakheni izithombe ezingaqaliswa kunoma iyiphi iseva noma indawo yamafu ngaphandle kwezinkinga zokuncika ezingalindelekile.
I-Docker ikuvumela ukuthi uchaze konke okudingayo ukuze usebenzise uhlelo lwakho lokusebenza ku-Dockerfile : inguqulo ye-Python noma ye-Java, imitapo yolwazi, ukucushwa okuyisisekelo, nokuningi. Bese umane wakhe isithombe, usihlole endaweni, bese usebenzisa isitsha noma kuphi lapho kudingeka khona. Lokhu kunciphisa kakhulu isimo sakudala esithi "sisebenza kukhompyutha yami" futhi kusiza ukubambisana namaqembu e-DevOps.
KuNjiniyela Wedatha, kuvamile ukupakisha izinsizakalo zokungenisa, ama-API amamodeli, abasebenzi bokucubungula , noma imisebenzi yokuhlanganisa ama-container. Lezi ziqukathi zibe sezihlanganiswa kumapulatifomu afana nama-Kubernetes noma amanye ama-orchestrator, yize leso sinyathelo singase sifike kamuva.
Izincwadi zobuchwepheshe ezibhekisela kuzo kanye nemiphakathi zigcizelela ukuthi i-Docker isibe yikhono elibalulekile kulabo abasebenza ngokusetshenziswa kwamamodeli kanye namapayipi, ngoba ikuvumela ukuthi uphinde wenze kabusha izindawo, wenze ngokuzenzakalelayo ukuthunyelwa, kanye nengqalasizinda yenguqulo ngendlela efana nendlela osebenzisa ngayo ikhodi yenguqulo.
Amamodeli okukhiqiza: kusukela kusikripthi kuya ku-API nge-Flask noma i-FastAPI
Esinye isinyathelo esibalulekile kule ndlela, ikakhulukazi uma uvela emlandweni weSayensi Yedatha, ukufunda ukuveza amamodeli njengezinsizakalo zewebhu . Ukugcina nje i-pickle noma ifayela lokucushwa akusanele: udinga ukudala ama-API angadliwa amanye amaqembu noma izinhlelo zokusebenza.
Amafreyimu alula njengeFlask noma i-FastAPI afanele lokhu. Ngawo, ungasetha i-API ngemigqa embalwa yekhodi ethola idatha nge-POST, isebenzisa imodeli yakho, futhi ibuyisele isibikezelo ngefomethi ye-JSON. Lawa masevisi angahlanganiswa nezakhiwo ezinkulu noma imisebenzi yokusakaza.
Ukuhlanganisa leli khono ne-Docker kukuvumela ukuthi udale izitsha ezizimele ngemodeli yakho , ezilungele ukusetshenziswa kumapulatifomu ahlukene. Ngaphezu kwalokho, i-FastAPI ifaka ukuhlanganiswa okulula nama-schema e-OpenAPI kanye nemibhalo ezenzakalelayo yesitayela se-Swagger, okwenza impilo ibe lula kulabo abasebenzisa isevisi yakho.
Le ndlela iyisango eliya ezweni lama- MLOps , lapho kungagcini nje ngokufaka imodeli, kodwa futhi nokuqapha ukusebenza kwayo, ukwenza idatha ibe yinguqulo, ukuqeqesha kabusha ngokuzenzakalela, kanye nokuphatha yonke impilo ekukhiqizeni. Ngisho noma ukugxila kwakho njengoNjiniyela Wedatha kungeyona i-MLOps kuphela, kubalulekile ukuqonda lo mongo.
Umehluko phakathi kwemodeli ehlala unomphela encwadini yokubhalela kanye nemodeli esesiphethweni esiqinile nesiqashwe kahle mkhulu kakhulu ngokwenani lenkampani, futhi ubunjiniyela bedatha buphakathi nendawo yalolo shintsho.
Ifu njengendawo yemvelo yoNjiniyela Wedatha
Namuhla, amapulatifomu amaningi edatha akhiwe kumhlinzeki wefu lomphakathi , ikakhulukazi i-AWS, i-Google Cloud, noma i-Azure. Ukuze uqedele umsebenzi wakho, kubalulekile ukutshala imali ekufundeni okungenani uhlelo olulodwa lwe-ecosystem ngokujulile.
Inketho yokuqala ethokozisayo yinhlanganisela yeDatabricks + Apache Spark , ikakhulukazi uma usuvele usebenzisa iPySpark. I-Databricks inikeza indawo ephethwe ngamaqoqo asatshalaliswa, ama-notebook abambisanayo, kanye namathuluzi amaningi agxile kubunjiniyela bedatha kanye nokufunda komshini. Ukuqonda kahle le nhlanganisela kuvula iminyango eminingi ezinkampanini ezinedatha eningi.
Enye inketho elula, ewusizo kuma-prototypes, ukuhlanganisa i-MongoDB namathuluzi afana ne-Streamlit , lapho ungagcina khona idatha ehlelekile kancane ku-MongoDB futhi wakhe amadeshibhodi noma izinhlelo zokusebenza zedatha ezisheshayo kakhulu nge-Streamlit ngaphandle kwengqalasizinda eyengeziwe.
Uma ufuna ukusebenzisa indlela esetshenziswa kakhulu efwini, ungagxila kumasevisi e-AWS noma e-GCP afana ne-Kinesis, i-Lambda, i-API Gateway, i-Pub/Sub, i-Dataflow, i-BigQuery, noma amathuluzi afanayo, akuvumela ukuthi wakhe imisebenzi yokusebenza engenaseva kanye nezakhiwo ezingandiswa kusukela ekuqaleni. Ezimweni eziningi, izinkampani ezinkulu ziyazisa kakhulu ulwazi lwangempela ngalezi zinsizakalo.
Abahlinzeki abanjengo-Google Cloud banikeza izindlela zokufunda ze-Data Engineer ezithile , ngamaqoqo ezifundo ezifunwayo, amalebhu asebenzayo, amabheji amakhono, kanye nokulungiselela izitifiketi ezisemthethweni. Ngokulandela indlela enjalo, ungahlela ukufunda kwakho futhi ulinganise intuthuko yakho uze ulungele ukuthatha isivivinyo.
Izinsizakusebenza, izindawo zokugcina impahla, kanye nendlela yokuzijwayeza ngempumelelo
Umbuzo ovame kakhulu kulabo abaqala le ndlela ukuthi yiziphi izinsiza abangazikhetha nokuthi yimaphi amaphrojekthi abazowenza ukuze ukufunda kungahlali kungokwemfundiso nje. Namuhla, kunezindawo zokugcina umphakathi ngesiSpanishi ezinemiqondo, izinselele zobuchwepheshe, kanye nokuqoqwa kwezinto zamahhala ezingasebenza njengesiqondiso esiphilayo.
Lezi zinqolobane zivame ukuhlukanisa izinsiza ngokwezinga (abaqalayo, abaphakathi, abathuthukile) kanye nolimi ukuze zikusize unqume ukuthi yini ozoyibuka kuqala. Nakuba okuqukethwe okuningi kusesiNgisini, ungasebenzisa njalo inketho ethi "translate to Spanish" yesiphequluli sakho noma usebenzise imibhalo engezansi kanye nokubhalwa okuzenzakalelayo kumavidiyo.
Ezinye izibonelo zemikhuba ewusizo zifaka phakathi izinselele "zezinsuku eziyi-100 zobunjiniyela bedatha" , lapho uzibophezela ukuchitha isikhathi esithile usuku ngalunye wakha okuthile: ipayipi elincane, iskripthi sokuhlanza idatha, imodeli yedatha, isixhumi se-API, njalo njalo. Ukuvumelana kuvame ukusebenza kangcono kunokuqhuma komsebenzi ngezikhathi ezithile.
Kunconywa kakhulu futhi ukufunda izincwadi namaphethini okuklama agxile kubunjiniyela bedatha , yize eziningi zingesiNgisi. Zifundisa izindlela eziqinisekisiwe zokuklama izinhlelo eziqinile, zikuveze ekwakhiweni kwezakhiwo zangempela, futhi zikusize ugweme amaphutha avamile abaqalayo.
Uma uthola okuthile okuwusizo ngempela, cabanga ngokufaka isandla kulezo zinqolobane ngokuthuthukiswa, ukuhumusha, izinsiza ezintsha, noma ukulungiswa kwamaphutha. Ukubamba iqhaza kumaphrojekthi avulekile akugcini nje ngokukusiza ukuthi ufunde kodwa futhi kuthuthukisa iphothifoliyo yakho yomphakathi nabaqashi abangaba khona.
Ukusesha umsebenzi, ukulungiselela inhlolokhono kanye nemibuzo evame ukubuzwa
Ezigabeni zokugcina zohambo, sekuyisikhathi sokugxila endleleni yokwethula iphrofayela yakho emakethe . Lokhu kuhlanganisa ukupholisha i-CV yakho, ukudala iphothifoliyo yamaphrojekthi edatha, ukugcina iphrofayela esebenzayo ezinkundleni zobuchwepheshe, kanye nokuzijwayeza izingxoxo zobuchwepheshe eziqondene nonjiniyela beDatha.
Izinkampani ziyazisa kakhulu ulwazi olusebenzayo kanye namaphrojekthi omuntu siqu abonisa ngokucacile inkinga oyixazululile, izinqumo zobuchwepheshe ozenzile, ubuchwepheshe obusebenzisile, kanye nemiphumela oyitholile. Ulwazi lwangaphambilini njengoNjiniyela Wedatha aludingeki; iphrojekthi yomuntu siqu ebhalwe kahle ingenza umehluko omkhulu.
Ngokuphathelene nemibuzo evame ukubuzwa, kuvela okufanayo njalo: ukuthi yimaphi amakhono obuchwepheshe okufanele uwabeke phambili , ukuthi kufanelekile yini ukufunda i-Spark noma ukuthi ama-Panda ne-SQL anele, ukuthi kufanelekile yini ukutshala isikhathi ekuqinisekisweni kwamafu, ukuthi kuthatha isikhathi esingakanani ukwenza ushintsho, noma ukuthi kungani abanye bethi i-Data Analyst "iphelelwe yisikhathi".
Ngokuphathelene namakhono, inhlanganisela ephumelelayo ivame ukuba uhlelo oluqinile, i-SQL ethuthukisiwe, ukuqonda okuqinile kwamamodeli edatha , ulwazi okungenani ngeplatifomu eyodwa yamafu, kanye nokuqonda okuyisisekelo kokuhlelwa nokusakaza. I-Spark iba yinto ebaluleke kakhulu lapho isebenza ngedatha eningi noma ezindaweni lapho isivele isetshenzisiwe khona.
Ngokuphathelene nezikhathi, isikhathi esidingekayo sokushintsha kusuka kusosayensi wedatha noma unjiniyela wedatha siye kunjiniyela wedatha siyahlukahluka, kodwa ngokuzinikela okuqhubekayo nokugxile , ungakulungela ukufaka izicelo zezikhundla zesikhashana noma zesikhashana ezinyangeni ezimbalwa. Isihluthulelo ukwakha isisekelo esiqinile, ugweme ukugxuma kusuka enkambweni uye kwenye ngaphandle kokuqeda noma iyiphi, futhi ugxile kumaphrojekthi abonisa amakhono akho.
Le ndlela eya kubunjiniyela bedatha ihlanganisa izisekelo zethiyori, ukujwayela okuningi kanye nomthamo omuhle welukuluku , kodwa ngokuzayo ivula iminyango kwenye yamaphrofayili afunwa kakhulu futhi abekwe kahle kakhulu emkhakheni wezobuchwepheshe, ngokwaneliseka okwengeziwe kokuqonda nokulawula lonke uhambo oluthathwa idatha enhlanganweni.
