- Indima yoNjiniyela Wedatha igxile ekuklameni nasekulondolozeni izinhlelo eziqoqa, ziguqule, futhi zigcine idatha ngendlela ethembekile nengalinganiswa.
- Indlela yokufunda ihlelwe ngamazinga: izinhlelo kanye nezizindalwazi, i-Big Data kanye namapayipi, bese ekugcineni ifu, ukuphepha kanye nokusakaza.
- Ukwazi i-SQL, ukumodela idatha, i-ETL, ukuhlelwa kwemisebenzi, izitsha, kanye nomhlinzeki wefu okungenani oyedwa kuyisihluthulelo sentuthuko yobungcweti.
- Amaphrojekthi asebenzayo, izindawo zokugcina ulwazi emphakathini, kanye nezitifiketi kusiza ukuhlanganisa ulwazi nokuthuthukisa izinketho zokufuna umsebenzi.
Indlela yomsebenzi wokuba uNjiniyela Wedatha Sekungomunye wemikhakha ekhanga kakhulu emhlabeni wedatha, ikakhulukazi kulabo abanezizinda ezifana ne-Data Analyst noma Idatha Sosayensi Futhi bafuna ukuthatha indlela yobuchwepheshe. Izinkampani eziningi zidinga abantu abakwazi ukuklama, ukwakha, nokugcina izinhlelo ezihambisa ulwazi, hhayi nje amamodeli okufunda komshini noma amadeshibhodi.
Ngesikhathi esifanayo, inani lezinsizakusebenza, izifundo kanye nezincomo Ulwazi olusakazwa ku-inthanethi lungaba lukhulu: ukuthi uqale nge-Python, noma uqale nge-SQL kanye nokubonisa, noma uye ngqo efwini noma e-Spark… Kulesi sihloko, uzothola indlela yokufunda ephelele ngesiSpanishi, esekelwe kokuqukethwe okubhekiselwe kukho futhi yandiswa ngomongo osebenzayo, ukuze wazi kahle ukuthi uqale kuphi, ukuthi ungathuthuka kanjani, nokuthi yiziphi izinqumo okufanele uzenze ekuthuthukisweni kwakho njengoNjiniyela Wedatha.
Uyini uNjiniyela Wedatha futhi kungani indima yakhe ichuma?
Un Unjiniyela Wedatha unesibopho sokuklama, ukwakha, kanye nokuqalisa Izinhlelo eziqoqa, ziguqule, zigcine, futhi zenze kutholakale idatha esetshenziswa yizinkampani ukwenza izinqumo. Ngenkathi usosayensi wedatha egxila kakhulu kumamodeli nasekuhlaziyweni, unjiniyela wedatha uqinisekisa ukuthi ulwazi lulethwa ngesikhathi, ngokuthembekile, ngokukhula, nangokuphephile.
Ngokwenzayo, umsebenzi wansuku zonke woNjiniyela Wedatha Ngokuvamile kufaka phakathi ukwakha amapayipi e-ETL noma e-ELT, ukuhlela izinqubo, kanye nokuklama ukwakheka kwedatha (amachibi edatha, izindawo zokugcina idatha, ama-datamarts), ukuhlanganiswa kwemithombo eminingi kanye nokubambisana namanye amaqembu afana nokuhlaziya, isayensi yedatha noma umkhiqizo.
Ngokusho kwemibiko ehlukahlukene yemboni, Isidingo Sonjiniyela Bedatha siyaqhubeka nokukhula Futhi imiholo yabo ngokuvamile iphakeme kuneyamaphrofayili esayensi yedatha ezimakethe eziningi, ngenxa yomthelela oqondile abanawo engqalasizinda yezobuchwepheshe kanye nekhono lenkampani lokusebenzisa idatha yayo.
Amapulatifomu agxile ekuqeqeshweni kwedatha aqokomisa lokho ngaphezu kwama-70% emisebenzi ethunyelwe ku-Data Engineer Zidinga ulwazi oluqinile lobunjiniyela besofthiwe kanye Amasistimu asabalalisiwenokuthi amazinga omholo ale ndima angadlula kalula lawo amanye amaphrofayili okuhlaziya lapho kuhlanganiswa amakhono okuhlela, amafu, kanye nokwakha.
Kusukela kuSosayensi Wedatha kuya kuNjiniyela Wedatha: kungani abaningi beshintsha
Ezinhlanganweni eziningi, ikakhulukazi izinkampani ezintsha noma izinkampani ezikhulayo, imingcele phakathi kwe-Data Scientist kanye ne-Data Engineer Azicaci nhlobo. Ngokuvamile, umuntu oqeqesha amamodeli kufanele futhi ahlanze idatha, akhe izikripthi zokukhipha, athuthe amafayela, enze izinqubo ngokuzenzakalela, futhi amise ngisho nama-API ukuze anikeze izibikezelo.
Uma uke wazithola wakha amapayipi, ukusabalalisa amamodeli "ngesandla" noma ukuxhuma imithombo yedatha eyinkulungwaneKungenzeka ukuthi usuvele usebenzisana eduze kakhulu nalokho okwenziwa yi-Data Engineer. Lokhu kuvezwa kobuchwepheshe kuvame ukuvusa isithakazelo sokuqonda yonke indlela yokusebenza, kusukela ekungenisweni kwedatha kuya ekukhiqizweni, futhi ungathembeli kakhulu kwamanye amaqembu noma izixazululo zesikhashana.
Isizathu esiyinhloko salolu shintsho yi- ukuzimela kwezobuchwephesheUma uqonda ukuthi amapulatifomu edatha aklanywe kanjani, ukuthi yibuphi ubuchwepheshe obukhona, nokuthi asetshenziswa kanjani efwini, ungaletha imibono yakho ekukhiqizweni ngendlela eqinile, ngaphandle kokunamathela kuma-notebook okuhlola angafinyeleli kumsebenzisi wokugcina.
Futhi, Imakethe yemisebenzi ifuna kakhulu amaphrofayili obunjiniyela bedathaNakuba izindima zesayensi yedatha kuphela zivame ukuzinza, isidingo sabantu sokwakha ingqalasizinda yedatha, amapayipi esikhathi sangempela, kanye nezinhlelo ezikwazi ukukhuliswa siyakhula, okwenza ushintsho lube yisinqumo esihlakaniphile seminyaka ezayo.
Amazinga omzila wobungcweti: abaqalayo, abaphakathi, nabathuthukile
Ukuze ugweme ukukhungathekiswa ulwazi oluningi kangaka, kuyasiza hlukanisa indlela ye-Data Engineer ibe amazinga amathathu amazinga okuvuthwa: osaqalayo, ophakathi nendawo, kanye nothuthukile. Umqondo awukona ukukudonsela phansi, kodwa ukukusiza ukuthi ubheke phambili ukuthi yini okufanele uyifunde kuqala ngokusekelwe endaweni oqala kuyo.
Ezingeni saqala Izisekelo zihlanganiswe ndawonye: ukuhlela, ukucabanga, ukulawula izinguqulo, kanye nezizindalwazi eziyisisekelo. Yilokhu okudingayo uma uqala kusukela ekuqaleni noma uvela esizindeni esingenabo ubuchwepheshe, njengendima egxile kakhulu ebhizinisini noma kumhlaziyi.
Ezingeni maphakathi Izihloko ezidingidwayo zifaka phakathi i-Big Data, amathuluzi okucubungula asatshalaliswa, ukwakheka kwamapayipi e-ETL, kanye nabahleli. Lapha uzoqala ukuhlola ubuchwepheshe ozobubona ezindaweni zokukhiqiza bese uqala ukucabanga njengomakhi wedatha.
Ezingeni kuthuthukile Amakhono efu, izitifiketi, ukuphepha, ukuthunyelwa okuqhubekayo, ukusakaza bukhoma, kanye nefu ngokwalo konke kufakiwe. ukufuna umsebenzi kanye nokulungiselela ingxoxo yobuchwephesheLesi yisigaba lapho uhlose khona ukuthola izikhundla eziphezulu noma ezikhethekile.
Njengomthetho wesithupha, uma Awukahleli kahle izinhlelo okwamanje.Kunengqondo ukuqala ngesigaba se-Programming and Databases. Uma usuvele ukhululekile nge-SQL kanye ne-Python ethile, ungagxumela ngokushesha ku-Big Data and Data Processing. Futhi uma umgomo wakho uwukuqinisekiswa kwefu, isigaba sefu sizoba yisihluthulelo.
Izisekelo zokuhlela kanye nokulawulwa kwenguqulo
Isisekelo cishe sayo yonke into kubunjiniyela bedatha yi- ukwazi ukuhlela ngokwahlulela okunengqondoAkukhona nje kuphela ukubhala izikripthi "ezisebenzayo," kodwa mayelana nokudala ikhodi elungisekayo, efundekayo, futhi elula ukuyilungisa. Kule ndawo, i-Python ivame ukuba yindawo engcono kakhulu yokungena ngenxa ye-syntax yayo elula kanye ne-ecosystem yayo enkulu kwisayensi yedatha kanye nobunjiniyela bedatha.
Kulesi sigaba kuyalulekwa ukuthi ucindezele kanzima imiqondo eyisisekelo yokuhlelaIzinhlobo zedatha, izakhiwo (uhlu, izichazamazwi, amasethi), imisebenzi, amakilasi, ukuphathwa kwamaphutha, kanye nokufunda nokubhala amafayela konke kuhlanganisiwe. Uma ukhetha ezinye izilimi ezifana neJava, iScala, i-R, noma iJulia, lezo nazo ziyasebenza, kodwa ezweni langempela lobunjiniyela bedatha, iPython neJava/Scala yizona ezihamba phambili.
Ngesikhathi esifanayo, kubalulekile ukufunda ukulawula inguqulo nge-GitAbaningi bakubona njengokuwusizo kuphela ekusebenzeni ngokubambisana, kodwa empeleni kukuvumela ukuthi ulandelele umlando wekhodi yakho, uqonde ukuthi yini eshintshile nokuthi nini, uvivinye imibono ngaphandle kokwesaba, futhi ugcine umsebenzi wakho uhlelekile. I-GitHub noma i-GitLab izoba yizinkundla zakho zansuku zonke zokubamba izindawo zokugcina nokubambisana.
Akudingeki ube yi-Git guru kusukela ngosuku lokuqala, kodwa uyadinga funda imiyalo eyisisekelo (init, engeza, commit, branch, merge, push, pull) kanye nokuqonda ukuthi amagatsha, izicelo zokudonsa, kanye nokubuyekezwa kwekhodi kusebenza kanjani. Le ndlela yokusebenza iyinto evamile kunoma yiliphi ithimba lobuchwepheshe elingabalulekile kangako.
Ama-database, i-SQL, kanye nokumodela kolwazi
Uma izisekelo zokuhlela sezimisiwe, sekuyisikhathi sokuhlola izizindalwazi kanye ne-SQLYilapho abantu abaningi bedideka khona ngokuhleleka: i-Python kuqala, bese kuba yi-SQL, noma okuphambene nalokho? Indlela enengqondo kakhulu ukuthuthuka ngesikhathi esifanayo, kodwa ukuqinisekisa ukuthi ukuphatha i-SQL kuba yinto yesibili kuwe.
Ngedatha ehlelekile, inketho enconywe kakhulu yile Ukuqala nge-PostgreSQLNgenxa yamandla ayo nangenxa yokuthi iyindinganiso evamile kumaphrojekthi amaningi. Uma usuvele ujwayelene ne-MySQL, i-SQLite, noma ezinye izinjini, izoqhubeka isebenza, yize i-PostgreSQL ivame ukunikeza ukuguquguquka okwengeziwe ezindaweni zobungcweti.
Kungumqondo omuhle futhi ukujwayelana Imininingwane ye-NoSQLnjenge-MongoDB yamadokhumenti noma i-Redis yamabhangqa e-key-value, kanye nezinye ezifana ne-Cassandra zamakholomu. Umqondo awukhona ukuwakhumbula wonke ngekhanda, kodwa ukuqonda izimo zokusetshenziswa kwawo, izinzuzo zawo kanye nokungalungi kwawo, nokwazi ukuthi kufanele ukhethe nini enye kunenye.
Yilapho-ke ukumodela idathaImodeli yobudlelwano, imodeli yobukhulu, imiqondo yamaqiniso nobukhulu, ukujwayela, okhiye abayinhloko nabangaphandle, ubuqotho bokubhekisela. Uzofunda ukucabanga ngokwezinhlelo zethebula, ubudlelwano, kanye nemibuzo ephumelelayo, okubalulekile kunoma yiluphi uhlaka lwesakhiwo olulandelayo.
Kamuva, uzongena ujule kakhulu amachibi edatha, izindawo zokugcina idatha, ama-data marts kanye nezindawo zokuthola idathaNgaphezu kwezindlela ezifana nokugcina ikholomu ngokumelene nomugqa, i-schema yezinkanyezi, i-schema yeqhwa, kanye namasu e-schema okufundwayo ngokumelene nokubhala, lokhu kuzokunikeza ulimi namaphethini asetshenziswa kumaphrojekthi wangempela ukuhlela ulwazi ngezinga.
Imiqondo ye-Big Data, i-analytics kanye ne-business intelligence
Ngokuqonda okucacile kwe-SQL kanye nezisekelo zedathabheyisi, kungumqondo omuhle ukubheka imiqondo ye-Big Data kanye nokuhlaziyaAkudingeki ube uchwepheshe kuzo zonke izinhlaka ohlelweni lwe-ecosystem, kodwa udinga ukuqonda ukuthi yiziphi izinkinga abazama ukuzixazulula nokuthi kungani zikhona.
Izwe le-Big Data lithembele ku- ukusatshalaliswa kokucubungulaKulo modeli, esikhundleni sokusebenzisa yonke into emshinini owodwa, umthwalo womsebenzi usatshalaliswa kuma-node amaningi. Amathuluzi afana ne-Apache Spark asethandwa kakhulu ekucubunguleni idatha enkulu, kokubili ku-batch kanye nokusakaza, futhi avame ukuba yingxenye yezinkampani zobuchwepheshe eziqhutshwa idatha.
Ngaphezu kwe-Big Data, kuyathakazelisa ukuthola umbono jikelele we- ubuhlakani bokwenziwa, ukufunda komshini kanye nobuhlakani bebhizinisiNakuba njengoNjiniyela Wedatha ngeke kudingeke uqeqeshe amamodeli ayinkimbinkimbi, kuzodingeka uwalungiselele idatha futhi uklame ingqalasizinda ezondla wona.
Uzobona nokuthi izinto zinjani Amathuluzi e-BI (i-Power BI, i-Tableau, i-Looker, njll.), izinqubo zokubika, kanye nezidingo zabahlaziyi bebhizinisi. Ukuqonda imisebenzi yabo kuzokusiza ukuthi uklame amapayipi edatha awusizo kakhulu namamodeli kulabo abasebenzisa ulwazi.
Ukucutshungulwa kwedatha: I-ETL, ukuhlelwa kanye nemibhobho yedatha
Inhliziyo yangempela yobunjiniyela bedatha ukuklama nokwakhiwa kwamapayipi edathaLapha uzofunda kahle ukuthi iyini i-ETL (Extract, Transform, Load), uma indlela ye-ELT inengqondo, ukuthi ungayihlela kanjani imisebenzi, uyiqaphe, futhi ululame ekuhlulekeni.
Ipayipi elijwayelekile lihlanganisa izigaba ukungenisa idatha kusuka emithonjeni eminingi (ama-API, izizindalwazi, amafayela, imigqa yemiyalezo), izinyathelo zokuhlanza nokuguqula (ukwenza kube ngokwejwayelekile, ukuhlanganiswa, ukucebisa) bese ekugcineni kulayishwa ohlelweni oluthile oluqondiwe, okungaba yindawo yokugcina idatha, ichibi ledatha, izizindalwazi ze-NoSQL noma ingxube yeziningana.
Kulesi simo, amathuluzi avela ukuze ukuhlelwa kokugeleza njenge-Apache Airflow noma ezinye izindlela zesimanje, ezikuvumela ukuthi uchaze ukuncika phakathi kwemisebenzi, ukuhlela ukwenziwa, ukulandelela lokho okuye kwenziwa, nokusabela emaphutheni. Nakuba inkampani ngayinye isebenzisa inqwaba ehlukile, indlela yokucabanga yokuhlela nokwenza izinqubo ngokuzenzakalela ivamile kubo bonke.
Iphuzu elibalulekile ikhathalogi yemiqondo evame ukusetshenziswa kulezi zindawo: imodeli yobudlelwano kanye nobukhulu, ichibi ledatha, imashi yedatha, indawo yokugcina idatha, umklamo wekholomu noma umugqa, izikimu zezinkanyezi nezeqhwakanye namasu okufunda nokubhala anezinhlelo ezahlukene. Ukuqonda okucacile kwaleli gama kuzokuvumela ukuthi uqonde imibhalo yobuchwepheshe, izincwadi ezikhethekile, kanye nemidwebo yezakhiwo.
Lesi sigaba singenye yalezo ezizuza kakhulu ekuzilolongeni okusebenzayo kanye namaphrojekthi amancane omuntu siqu, lapho ungakwenza khona yakha amapayipi asuka komunye nomunyenoma ngabe kumayelana nedatha yomphakathi, futhi uzijwayeze amaphethini ajwayelekile ozowabona kamuva ezindimeni zobungcweti.
Ukuphepha emipayipini yedatha kanye namapulatifomu
Isinyathelo sokuqala ukusebenzisa isimiso sokuthi ilungelo elincane kakhulu ezindimeni nasezimvumeniI-akhawunti ngayinye yesevisi, yomsebenzisi, noma yohlelo lokusebenza kufanele ibe nokufinyelela okudingekayo kuphela ukuze yenze umsebenzi wayo, hhayi okunye. Lokhu kunciphisa indawo yokuhlasela futhi kunciphise umthelela wamaphutha noma ukuvuza.
Kubalulekile futhi ukuqonda ukuthi isebenza kanjani ukubethela kwedatha kusendleleni nasekwephumulweniSebenzisa i-HTTPS, i-TLS kanye nezinqubo ezivikelekile lapho uhambisa idatha phakathi kwezinsizakalo, futhi unike amandla ukubethela kudathabheyisi, amabhakede okugcina noma ezinye izinhlelo lapho kugcinwa khona ulwazi.
Uma udalula ama-API noma izinsizakalo zamamodeli, kufanele unake imininingwane efana ne- ukuqinisekiswa nokugunyazwa (amathokheni, okhiye be-API, i-OAuth, njll.), khawulela ukufinyelela kuma-endpoints abalulekile, kanye nomsebenzi wesistimu wokungena ngemvume ukuze uhlole ukusetshenziswa kabi. Awudingi ukuba uchwepheshe wezokuphepha, kodwa udinga izinga elanele lobuchwepheshe ukuze wenze izinqumo ezifanele.
Konke lokhu akugcini nje ngokuvimbela ukwesabisa, kodwa futhi Qinisa iphrofayela yakho yobungcweti emehlweni enkampani, njengoba ubonisa ukuqaphela umthelela wangempela womsebenzi wakho ebhizinisini kanye nokuvikelwa kwedatha yamakhasimende kanye neyomsebenzisi.
Izinhlobo zomklamo wesakhiwo sokugcina kanye nedatha
Uma ushintsha kusuka ekusebenzeni ngamasethi edatha angaguquki njengososayensi wedatha ukuya ekubeni unjiniyela wedatha, kushintsha ngokuphelele ubudlelwano bakho nesitorejiAkusekho mayelana nokuvula i-CSV endaweni, kodwa mayelana nokuklama izinhlelo ezisekela ukugeleza kwedatha okuqhubekayo, ukushintsha ama-schema, kanye nabathengi abaningi ngasikhathi sinye.
Empilweni yakho yansuku zonke uzohlanganisa izinhlobo ezahlukene zesitoreji: imininingwane yolwazi (PostgreSQL, MySQL) ngolwazi oluhlelekile nolwezentengiselwano; Izizindalwazi ze-NoSQL njenge-MongoDB (amadokhumenti), i-Redis (inani lokhiye) noma i-Cassandra (amakholomu) ngezidingo ezithile zokusebenza, ukuguquguquka kweskimu noma ukukala okuvundlile.
Okungeziwe kulokhu yi- isitoreji sefu sezinto (i-Amazon S3, i-Azure Data Lake Storage, i-Google Cloud Storage), eseyiyona nto eyinhloko yamachibi amaningi edatha anamuhla. Inqwaba yedatha eluhlaza necutshungulwayo igcinwa lapha, ngokuvamile ngamafomethi afana ne-Parquet noma i-Avro, alungele ukusetshenziswa yizinjini ezahlukene zokuhlaziya.
Ukuklama izakhiwo zedatha zesimanje kuhilela ukucabanga ngakho indlela idatha egeleza ngayo Kusukela emthonjeni kuya kumthengi, yiziphi izingqimba eziphakathi zekhwalithi, ukubusa, noma ukuguqulwa okudingekayo, futhi konke lokhu kungahlelwa kanjani ukuze kugcinwe kahle? Ukwazi ukufunda nokudala imidwebo yokwakha kuzoba yingxenye evamile yomsebenzi wakho.
Ngaphezu kwalokho, izinhlangano eziningi zamukela izakhiwo ezigxile ekusakazeni, lapho ubuchwepheshe obufana Apache Kafka Badlala indima ehamba phambili njengomgogodla wemicimbi, okusiletha esigabeni esilandelayo.
Ukusakaza kanye nokucubungula ngesikhathi sangempela nge-Apache Kafka
Ukuhlaziywa kwedatha okuningi kwendabuko kwenziwe ngemodi ye-batch: ukulayisha idatha njalo, ukuyicubungula, bese ukhiqiza imiphumela.Kodwa-ke, izinkampani eziningi zidinga ukusabela ngesikhathi sangempela kulokho okwenzekayo, kusukela ekuthengiselaneni kwezezimali kuya emisebenzini yabasebenzisi noma izinzwa ze-IoT.
Kulesi simo, u-Apache Kafka uvela njengo ipulatifomu yokusakaza imicimbi Yamukelwa izinhlangano eziyizinkulungwane ezingamashumi emhlabeni jikelele, i-Kafka ivumela abasebenzisi ukuthi bashicilele futhi basebenzise imiyalezo ezihlokweni, ngabakhiqizi nabathengi abahlukanisiwe, futhi bakhulise uhlelo ukuze luphathe kusukela kwezigidi ezimbalwa kuya kwezigidi zemicimbi ngomzuzwana.
Kunjiniyela Wedatha, ukuqonda kahle Ukwakhiwa kukaKafka Imiqondo eyinhloko ihlanganisa: ukuthi yiziphi izihloko, izingxenye, abadayisi, abakhiqizi, abathengi, amaqembu abathengi, kanye ne-offset. Futhi, indlela yokuhlanganisa i-Kafka nezinhlelo ezingezansi (izizindalwazi, izindawo zokugcina idatha, izinhlelo zokuxwayisa) kanye nezinqubo zokuhlaziya zesikhathi sangempela.
Amamodeli amaningi okufunda komshini nawo aqala ukusebenza emifudlaneni yedatha, okuwaphoqa ukuthi ahlangane Ama-MLOp anamapulatifomu okusakaza bukhoma ukuletha izibikezelo ezibukhoma. I-Kafka ayisekho nje "ebunye ubuchwepheshe" futhi iba yisisekelo sezakhiwo zesimanje ezigxile emicimbini.
Abaphathi be-IT ezinkampanini ezinkulu babheka izinhlelo zokusakaza njengezinhlelo zokusebenza ingxenye ebalulekile yamasu abo edatha kanye ne-AIUkubika intuthuko ebalulekile ngokubuyiselwa kokutshalwa kwezimali lapho usebenzisa lezi zakhiwo. Ukufunda i-Kafka kanye nemibono ehlobene kukubeka phambili kunabaningi abazongenela ukhetho.
Ama-Container, i-Docker, kanye nokuthunyelwa kwesevisi
Ekushintsheni kusuka ekubeni usosayensi wedatha kuya ekubeni unjiniyela wedatha, iphuzu lokuguquka liwukukwazi Ukupakisha nokuthumela izinsizakalo nge-DockerUqala ukusebenzisa izikripthi emshinini wakho uye ekwakheni izithombe ezingaqaliswa kunoma iyiphi iseva noma indawo yamafu ngaphandle kokumangala kokuthembela.
I-Docker ikuvumela ukuthi uchaze ku-Dockerfile Konke okudingayo ukuze usebenzise uhlelo lwakho lokusebenzaInguqulo ye-Python noma ye-Java, imitapo yolwazi, ukucushwa okuyisisekelo… Ngemuva kwalokho kufanele wakhe isithombe, usivivinye endaweni, bese usebenzisa isitsha noma kuphi lapho kudingeka khona. Lokhu kunciphisa kakhulu isimo sakudala esithi “sisebenza kukhompyutha yami” futhi kwenza kube lula ukusebenzisana ne-DevOps.
KuNjiniyela Wedatha, kuvamile ukupakisha izinsizakalo zokungenisa, ama-API emodeli, abasebenzi bokucubungula noma imisebenzi yokuhlelwa kweqembu equkethe amakhonteyina. Lawa makonteyina abe esehlanganiswa kumapulatifomu afana namaKubernetes noma amanye ama-orchestrator, yize leso sinyathelo singase sifike kamuva.
Izincwadi ezibhekisela kuzo kanye nemiphakathi yezobuchwepheshe zigcizelela ukuthi I-Docker isibe yikhono elibaluleke kakhulu Kulabo abasebenza ngokufakwa kwamamodeli kanye namapayipi, ngoba kukuvumela ukuthi ukhiqize kabusha izindawo, wenze ngokuzenzakalelayo ukuthunyelwa, kanye nengqalasizinda yenguqulo ngendlela efana nendlela okhokha ngayo ikhodi yenguqulo.
Amamodeli okukhiqiza: kusukela kusikripthi kuya ku-API nge-Flask noma i-FastAPI
Esinye isithiyo esibalulekile kule ndlela, ikakhulukazi uma uvela kwi-Data Science, ukufunda Ukuveza amamodeli njengezinsizakalo zewebhuAkusanele ukulondoloza i-pickle noma ifayela lokucushwa: Ama-API kumele adalwe angadliwa amanye amakhompyutha noma izinhlelo zokusebenza.
Izinhlaka ezilula ezifana I-Flask noma i-FastAPI Zilungele lokhu. Ngazo, ungasetha i-API ngemigqa embalwa nje ethola idatha nge-POST, isebenzisa imodeli yakho, futhi ibuyisele isibikezelo ngefomethi ye-JSON. Lezi zinsizakalo zingahlanganiswa nezakhiwo ezinkulu noma ukugeleza kokusakaza.
Ukuhlanganisa leli khono ne-Docker kukuvumela ukuthi udale izitsha ezizimele ezinemodeli yakhoIlungele ukusetshenziswa kumapulatifomu ahlukahlukene. Ngaphezu kwalokho, i-FastAPI ifaka ukuhlanganiswa okulula nama-schema e-OpenAPI kanye namadokhumenti ezenzakalelayo esitayela se-Swagger, okwenza impilo ibe lula kulabo abasebenzisa isevisi yakho.
Le ndlela iyisango eliya ezweni le- MLOpsLokhu akubandakanyi nje kuphela ukufaka imodeli, kodwa futhi nokuqapha ukusebenza kwayo, ukufaka idatha yenguqulo, ukuqeqesha kabusha ngokuzenzakalela, kanye nokuphatha yonke impilo ekukhiqizeni. Ngisho noma ukugxila kwakho njengoNjiniyela Wedatha kungeyona i-MLOps kuphela, ukuqonda lo mongo kubalulekile.
Umehluko phakathi kwemodeli ehlala unomphela kukhompyutha ephathekayo kanye nemodeli etholakala endaweni yokugcina eqinile neqashwe kahle mkhulu kakhulu ngokwenani lenkampani, futhi Ubunjiniyela bedatha buphakathi nendawo yalolo shintsho.
Ifu njengendawo yemvelo yoNjiniyela Wedatha
Namuhla, amapulatifomu amaningi edatha akhelwe phezu kwawo umhlinzeki othile wamafu omphakathiIkakhulukazi i-AWS, i-Google Cloud, noma i-Azure. Ukuze uqedele umsebenzi wakho, kubalulekile ukuzibophezela ekufundeni okungenani uhlelo olulodwa lwe-ecosystem ngokujulile.
Inketho yokuqala ethokozisayo inhlanganisela I-Databricks + i-Apache SparkIkakhulukazi uma usuvele ujwayelene ne-PySpark. I-Databricks inikeza indawo ephethwe ngamaqoqo asatshalaliswa, ama-notebook asebenzisanayo, kanye namathuluzi amaningi agxile kubunjiniyela bedatha kanye nokufunda komshini. Ukuqonda kahle le nhlanganisela kuvula iminyango eminingi ezinkampanini ezinedatha eningi.
Enye inketho elula, ewusizo kuma-prototypes, ukuhlanganisa I-MongoDB enamathuluzi afana ne-Streamlitlapho ungagcina khona idatha ehlelekile kancane ku-MongoDB futhi wakhe amadeshibhodi noma izinhlelo zokusebenza zedatha ezisheshayo kakhulu nge-Streamlit ngaphandle kwengqalasizinda eyengeziwe.
Uma ufuna indlela "etholakala efwini", ungagxila kuyo Izinsizakalo ze-AWS noma ze-GCP njengeKinesis, iLambda, i-API Gateway, iPub/Sub, iDataflow, iBigQuery, namathuluzi afanayo, akuvumela ukuthi wakhe imisebenzi yokusebenza engenaseva kanye nezakhiwo ezingandiswa kusukela ekuqaleni. Ezimweni eziningi, izinkampani ezinkulu ziyazisa kakhulu ulwazi lwangempela ngalezi zinsizakalo.
Abahlinzeki abanjengokunikezwa kwe-Google Cloud Izindlela zokufunda eziqondene nonjiniyela wedathaNgamaqoqo ezifundo ezifunwa kakhulu, amalebhu asebenzayo, amabheji amakhono, kanye nokulungiselela izitifiketi ezisemthethweni, le ndlela yokufunda ikuvumela ukuthi uhlele ukufunda kwakho futhi ulandelele inqubekela phambili yakho uze ulungele ukubhala isivivinyo sakho.
Izinsizakusebenza, izindawo zokugcina impahla, kanye nendlela yokuzijwayeza ngempumelelo
Umbuzo ovame kakhulu kulabo abaqala lo mzila uthi ukuthi yiziphi izinsiza okufanele uzikhethe kanye nokuthi yiziphi imiklamo okufanele uyenze Ukuze ukufunda kungahlali kungokwemfundiso nje kuphela. Namuhla, kunezindawo zokugcina umphakathi ngesiSpanishi ezinemiqondo, izinselele zobuchwepheshe, kanye nokuqoqwa kwezinto zamahhala ezingasebenza njengesiqondiso esiphilayo.
Kulezi zindawo zokugcina, izinsiza zivame ukuphawulwa ngokuthi izinga (oqalayo, ophakathi, othuthukile) Futhi ngolimi, ukukusiza ukuthi unqume ukuthi yini ozoyibuka kuqala. Nakuba okuqukethwe okuningi kungokwesiNgisi, ungasebenzisa njalo inketho ethi "translate to Spanish" yesiphequluli sakho noma usebenzise imibhalo engezansi ezenzakalelayo kanye nemibhalo eqoshiwe kumavidiyo.
Ezinye izibonelo zemikhuba ewusizo zifaka phakathi izinselele ezifana "nezinsuku eziyi-100 zobunjiniyela bedatha"lapho uzibophezela khona ekunikeleni isikhathi esithile nsuku zonke ekwakheni okuthile: ipayipi elincane, iskripthi sokuhlanza, imodeli yedatha, isixhumi se-API, njalo njalo. Ukuvumelana kuvame ukukhokhela okungaphezu komsebenzi oqhubekayo ngezikhathi ezithile.
Kunconywa kakhulu futhi ukuthi ufunde izincwadi kanye namaphethini okuklama aqondiswe kubunjiniyela bedathaNakuba eziningi zikhuluma isiNgisi, zifundisa izindlela eziqinisekisiwe zokuklama izinhlelo eziqinile, zikuveze ekwakhiweni kwezakhiwo zangempela, futhi zikusize ugweme amaphutha avamile abaqalayo.
Uma uthola okuthile okuwusizo ngempela, cabanga ngakho faka isandla kulezo zinqolobane zokugcina ngokuthuthukiswa, ukuhumusha, izinsizakusebenza ezintsha, noma ukulungiswa. Ukubamba iqhaza kumaphrojekthi avulekile akusizi nje kuphela ukuthi ufunde, kodwa futhi kuthuthukisa iphothifoliyo yakho yomphakathi nabaqashi abangaba khona.
Ukusesha umsebenzi, ukulungiselela inhlolokhono kanye nemibuzo evame ukubuzwa
Ezigabeni zokugcina zalo mzila, sekuyisikhathi sokugxila ku- Indlela yokwethula iphrofayela yakho emaketheLokhu kuhlanganisa ukupholisha i-CV yakho, ukudala iphothifoliyo yamaphrojekthi edatha, ukugcina iphrofayili esebenzayo kumapulatifomu ochwepheshe, kanye nokuzijwayeza izingxoxo zobuchwepheshe eziqondene nonjiniyela beDatha.
Izinkampani zivame ukukwazisa kakhulu. ulwazi olusebenzayo kanye namaphrojekthi akho lapho kusobala khona ukuthi iyiphi inkinga oyixazululile, yiziphi izinqumo zobuchwepheshe ozenzile, ubuchwepheshe obusebenzise, kanye nemiphumela oyitholile. Akudingeki ukuthi usebenze njengoNjiniyela Wedatha ngaphambili; iphrojekthi yomuntu siqu enhle nebhalwe kahle ingenza umehluko omkhulu.
Ngokuphathelene nemibuzo evame ukubuzwa, kuvela okufanayo njalo: ukuthi yimaphi amakhono obuchwepheshe okufanele abekwe phambiliKungakhathaliseki ukuthi kuyafaneleka yini ukufunda i-Spark noma i-Pandas kanye ne-SQL kwanele, kungakhathaliseki ukuthi kuyafaneleka yini ukutshala isikhathi ekuqinisekisweni kwamafu, ukuthi kuthatha isikhathi esingakanani ukwenza ushintsho, noma kungani abanye bethi i-Data Analyst "iphelelwe yisikhathi".
Ngokuphathelene namakhono, inhlanganisela ephumelelayo ivame ukuba izinhlelo eziqinile, i-SQL ethuthukisiwe, izisekelo zokumodela idathaUlwazi lokuphatha okungenani ipulatifomu eyodwa yamafu kanye nokuqonda okuyisisekelo kokuhlelwa nokusakaza kubalulekile. I-Spark iba yinto ebaluleke kakhulu lapho isebenza ngedatha eningi noma ezindaweni lapho isivele isetshenzisiwe khona.
Ngokuphathelene nezikhathi zesikhathi, isikhathi esidingekayo sokushintsha kusuka kusosayensi wedatha noma unjiniyela wedatha siye kuNjiniyela Wedatha siyahlukahluka, kodwa nge ukuzinikela okuqhubekayo nokugxile kahleEzinyangeni ezimbalwa ezizayo, ungase ulungele ukufaka izicelo zezikhundla zesikhashana noma zesikhashana. Into ebalulekile ukwakha isisekelo esiqinile, ugweme ukugxumagxuma usuka kwenye indlela uye kwenye ungaqedi lutho, futhi ugxile kumaphrojekthi abonisa amakhono akho.
Le ndlela eya kubunjiniyela bedatha ihlanganisa izisekelo zemfundiso, ukuzijwayeza okuningi, kanye nomthamo omuhle wokufuna ukwaziKodwa-ke, kuvula iminyango kwenye yamaphrofayili afunwa kakhulu futhi abekwe kahle kakhulu emkhakheni wezobuchwepheshe, ngokwaneliseka okwengeziwe kokuqonda nokulawula lonke uhambo oluthathwa yidatha enhlanganweni.
