I-Apache Kafka: Kuyini, ukuthi isebenza kanjani, nokuthi kungani ibalulekile kudatha enkulu

Informatec Digital » Aplicaciones » I-Apache Kafka: Kuyini, ukuthi isebenza kanjani, nokuthi kungani ibalulekile kudatha enkulu

Ukuphatha nokucubungula inani elikhulu ledatha ngesikhathi sangempela kuyinselele izinkampani eziningi ezibhekene nayo namuhla. Njalo ngomzuzu, emikhakheni eyahlukahlukene njenge-e-commerce, ezokuxhumana, kanye nebhange, kukhiqizwa izigidi zemicimbi kanye nezingcezu zolwazi ezidinga ukuphathwa kahle, ngokuguquguqukayo, nangokushesha. Kulesi simo, kuvele ubuchwepheshe obushintshashintshayo obuvumela izinkampani ukuthi ziguqule lokhu kugeleza kwedatha kube amathuba okwenza ngcono izinqubo, zithuthukise ulwazi lwamakhasimende, futhi zenze izinqumo ezinolwazi ngokushesha. Esinye sezixazululo ezivelele nezisetshenziswa kabanzi kule ndawo yi-Apache Kafka.

I-Apache Kafka isuke ekubeni ithuluzi elilula lokuthumela imiyalezo yaba yipulatifomu ehamba phambili yokusakaza idatha, kokubili ezindaweni zebhizinisi kanye namaphrojekthi edatha enkulu yanoma yiluphi usayizi. Lesi sihloko sizochaza ngokuningiliziwe ukuthi iyini i-Apache Kafka, ukuthi isebenza kanjani, izinhlelo zayo zokusebenza, izinzuzo zayo, nokuthi kungani isibe yindinganiso yokudlulisa idatha ngesikhathi sangempela kanye nokuphathwa kwayo.

Iyini i-Apache Kafka?

I-Apache Kafka iyipulatifomu evulekile, esatshalaliswayo eyenzelwe ukucubungula, ukugcina, nokudlulisa inani elikhulu ledatha ngesimo semicimbi noma imiyalezo phakathi kwezinhlelo ezahlukene, izinhlelo zokusebenza, noma izinsizakalo. Ekuqaleni yadalelwa ukuphatha ukugeleza kwedatha okukhulu kwe-LinkedIn, i-Kafka yanikelwa kwi-Apache Software Foundation, futhi kusukela ngaleso sikhathi, intuthuko yayo ithuthuke ngokushesha, yandisa ukusetshenziswa kwayo ngale kwenhloso yayo yokuqala.

I-Apache Kafka ingachazwa njengohlelo lokuthumela imiyalezo olubhalisa ngokushicilela olukwazi ukusingatha izigidi zemicimbi ngomzuzwana ngokubambezeleka okuphansi, ukubekezelela amaphutha aphezulu, kanye nokukhula okuvundlile . Okuhlukanisa i-Kafka nezinye izixazululo yikhono layo lokusebenza ngemifudlana yedatha "engenamkhawulo"—okungukuthi, idatha engenasiqalo noma isiphetho esichaziwe—futhi iqinisekisa ukucutshungulwa kwesikhathi sangempela.

Namuhla, izinkampani ezihamba phambili njengeNetflix, i-Uber, i-LinkedIn, i-Spotify, i-PayPal, i-Cisco, i-Oracle, i-Twitter, ne-Adidas zithembele ku-Apache Kafka ukuze zisekele imisebenzi yazo ebalulekile futhi ziphathe ulwazi oluningi.

Isetshenziselwa ini i-Apache Kafka?

Umsebenzi oyinhloko ka-Apache Kafka ukusebenza njengomgwaqo omkhulu wedatha, okwenza ulwazi luhambe ngokushesha phakathi kwamaphuzu noma izinhlelo ezahlukene ngaphakathi kwenhlangano. Luvumela ukuqoqwa, ukucubungula, kanye nokugcinwa kwedatha evela emithonjeni eminingi futhi lwenze lutholakale kuzinhlelo zokusebenza eziludingayo, kungaba ukuhlaziywa, ukuqapha, ukuhlanganiswa, noma ukwenza ngokuzenzakalelayo kwenqubo.

Okunye okuvame ukusetshenziswa kanye namacala okufaka isicelo e-Apache Kafka afaka:

• Ukucubungula umcimbi wesikhathi sangempela: kusukela ekuhlaziyweni kokuchofoza iwebhusayithi kuya ekuqashweni kwedivayisi ye-IoT.
• Ukudluliswa kwedatha phakathi kwama-microservices noma amasistimu: Isebenza njengesixhumanisi esingasindi nesiqinile sokuxhumana namamojula ahlukene wezakhiwo ezisabalalisiwe.
• Ukuphathwa kwamapayipi edatha: isiza ukungeniswa, ukuguqulwa nokugcinwa kwedatha enkulu, ngokwesibonelo, kudatha enkulu noma amaphrojekthi wobuhlakani bokwenziwa.
• Inqubo Yebhizinisi Okuzenzakalelayo: ikuvumela ukuthi uqalise izenzo ezizenzakalelayo noma nini uma kwenzeka umcimbi ohlobene.
• Ukuhlanganiswa kwezinhlelo zefa nezinhlelo zokusebenza ezintsha zamafu: I-Kafka ingavala igebe phakathi kobuchwepheshe obuhlukahlukene nendawo.

Ukwakheka kwayo okuguquguqukayo kuholele ekutheni iKafka isetshenziswe emikhakheni eyahlukahlukene njenge-e-commerce, ezezimali, ezokuthutha, ezempilo, ezokuxhumana noma embonini yokukhiqiza , yaba yingxenye ebalulekile yokuguqulwa kwedijithali kanye nomhlaba we-big data.

Isebenza kanjani i-Apache Kafka?

I-Apache Kafka isebenza ngokwakhiwa okusatshalaliswe kanye nokwe-modular okwenzelwe ukuphatha idatha enkulu ngokwethembeka, ngokukala, nangokubekezelela amaphutha. I-Kafka isetshenziswa futhi isebenza kuseva eyodwa noma ngaphezulu eyakha i- cluster . Iseva ngayinye ngaphakathi kwaleli cluster ibizwa ngokuthi "umthengisi."

Imilayezo noma imicimbi e-Kafka ihlelwe ngezihloko kanye nokwahlukaniswa. Cabanga "ngesihloko" njengesiteshi lapho kushicilelwa khona imiyalezo yesigaba esifanayo (isibonelo, ukuthengiselana kwebhange, imicimbi yenzwa, amalogi okufinyelela, njll.). "Isihloko" ngasinye sihlukaniswe "ngama-partitions," asabalalisa umthwalo futhi enze kube lula ukufana. Lesi sakhiwo sivumela abakhiqizi abaningi nabathengi ukuthi basebenze ngesikhathi esisodwa, benikeza ukukhuliswa okungenamkhawulo.

Ngaphakathi kwesigaba ngasinye, i-Kafka iqinisekisa ukuhleleka kwemiyalezo ; okungukuthi, imiyalezo igcinwa futhi ilandwe ngendlela efanayo naleyo ekhiqizwe ngayo. Umlayezo ngamunye uhlotshaniswa nesihluthulelo futhi ungagcinwa isikhathi esingalungiselelwa, okuvumela abathengi ukuthi bafunde imicimbi ngesikhathi sangempela noma kusukela endaweni ethile emlandweni wedatha.

Isakhiwo se-Kafka sihlanganisa izinhlobo eziningana zezingxenye ezibalulekile:

• Abakhiqizi: Lezi izinhlelo zokusebenza noma amasistimu athumela imilayezo ku-Kafka, eshicilela ulwazi esihlokweni esisodwa noma ngaphezulu.
• Abathengi: Lezi yizinhlelo zokusebenza ezifunda imilayezo evela ezihlokweni futhi ziyicubungule ngokwezidingo zazo.
• Abathengi: Amaseva akha iqoqo le-Kafka futhi alawula ukugcinwa nokusatshalaliswa kwemilayezo phakathi kwama-partitions.
• Ama-partitions nama-replicas: Isihloko ngasinye sihlukaniswe ngama-partitions, aphinde aphindaphindwe kuwo wonke ama-broker amaningi ukuze kuqinisekiswe ukutholakala nokubekezelela amaphutha.

Eminyakeni yamuva nje, iKafka ishintshe ukwenza kube lula ukuphathwa kwayo nokunciphisa ukuncika kwangaphandle. Isibonelo, ekuqaleni yayisebenzisa i-Apache ZooKeeper njengohlelo lokuxhumanisa ukugcina ukuvumelanisa phakathi kwabathengisi, kodwa ngokufika kwemodi ye-KRaft (i-Kafka Raft Protocol), leyo ndima idluliselwe kubathengisi beKafka uqobo, okwenza ukwakheka kube okuqinile futhi kube lula.

Izici ezibalulekile ze-Apache Kafka

Izici eziyinhloko ezenze i-Kafka yaba ubuchwepheshe obuyisethenjwa bokucubungula ukusakazwa kwedatha yilezi:

• Ukusebenza okuphezulu kakhulu: I-Kafka iyakwazi ukuphatha izigidi zemilayezo ngomzuzwana ngokubambezeleka okuncane, ilungele izinhlelo zokusebenza zesikhathi sangempela.
• Ukukala okuvundlile: Kulula ngendlela emangalisayo ukwengeza ama-node amasha kuqoqo lakho ukuze ukhuphule umthamo wokucubungula nokugcina.
• Ukubekezelela amaphutha nokutholakala okuphezulu: Ngenxa yokuphindaphinda kwe-partition kanye nesakhiwo esabalalisiwe, i-Kafka ingaqhubeka nokusebenza ngisho nalapho amanye amaseva ayo ehluleka.
• Ukuvumelana nezimo ekugcineni idatha: Izihloko zingalungiselelwa ukugcina imilayezo isikhathi esithile noma ingunaphakade.
• Amandla okusetshenziswa okuningi: Abathengi abaningi bangakwazi ukufunda idatha efanayo, bevumela ukuhlaziya okufanayo, ukuqapha, nokucubungula izimo.
• Ukuhlanganiswa okulula: I-Kafka inikeza ama-API ngezilimi ezihlukahlukene futhi ixhuma kalula nobunye ubuchwepheshe obufana ne-Hadoop, Spark, Flink, kanye nezinsizakalo zamafu.

Ngaphezu kwalokho, i-Kafka inama-API amaningana ukumboza izidingo ezahlukene:

• I-Producer API: Ivumela izinhlelo zokusebenza ukushicilela ukusakazwa kwelogi.
• I-Consumer API: Ikuvumela ukuthi ubhalisele futhi ucubungule amarekhodi ashicilelwe.
• I-Connector API: Isiza ukungenisa nokuthekelisa kwedatha ngamasistimu angaphandle.
• I-Streams API: Ikhethekile ekucubunguleni ukusakaza ngaphakathi kwe-Kafka ngokwayo, ilungele ukudala ukuhlaziya kwesikhathi sangempela izinhlelo zokusebenza ze-Java.
• I-Admin API: Ukuphatha nokuphatha ama-broker, izihloko kanye nezinto zeqoqo.

Ukuqhathanisa phakathi kwe-Apache Kafka nezinye izixazululo zemiyalezo

Umbuzo ovelayo ukuthi i-Kafka ihluke kanjani kwezinye izixazululo ezidumile njenge-RabbitMQ noma izinhlelo ezisekelwe ku-AMQP. Nawa amaphuzu abalulekile:

• Imodeli yedatha: I-Kafka isebenzisa imodeli yelogi ehlukanisiwe esekelwe ezihlokweni nama-partitions, isekela amaphethini wokubhalisa kokushicilela kanye nolayini, kuyilapho i-RabbitMQ igxile ekulawuleni imilayezo yakudala.
• Ukugcinwa komlayezo: E-Kafka, imilayezo ingalungiselelwa ukuthi igcinwe isikhathi esithile noma ingunaphakade, kuyilapho ezinhlelweni ezifana ne-RabbitMQ ivamise ukususwa ngemva kokusetshenziswa.
• Ukukala nokusebenza: I-Kafka yakhelwe ukukala ngaphandle komthungo ngenani elikhulu ledatha ngenxa yesakhiwo sayo esabalalisiwe, kuyilapho ezinye izindlela zingadinga ukulungiswa kabusha okuyinkimbinkimbi.
• Ukusekelwa kwabathengi abaningi: I-Kafka ivumela abathengi abaningi ukuthi bacubungule imilayezo efanayo ngokuhambisana, okulungele ukuhlaziya kwesikhathi sangempela noma ukuhlolwa.
• Izimiso zokuxhumana: I-Kafka isebenzisa iphrothokholi kanambambili ngaphezu kwe-TCP elungiselelwe ukusetshenziswa kwayo, kuyilapho ezinye izinhlelo zingase zidinge izengezo ukuze zisekele amaphrothokholi ahlukene.

Lokhu kuvumelana nezimo nokuqina kuchaza ukuthi kungani i-Apache Kafka iyisinqumo esikhethwayo samaphrojekthi amakhulu wedatha kanye nezakhiwo zesimanje zezinsizakalo ezincane.

Izinzuzo Zokuncintisana ze-Apache Kafka

Uma ibhizinisi lakho noma iphrojekthi idinga ukuphatha imiqulu emikhulu yolwazi, i-Kafka inikeza izinzuzo eziphawuleka ngempela:

• I-scalability engenamandla: Vele ungeze abathengisi abasha ukuze ukhulise iqoqo lakho njengoba udinga.
• Ukubambezeleka okuphansi (ama-millisecond nje): Ukubambezeleka ekudluliselweni komlayezo kuncane, okuvumela isenzo esisheshayo kudatha.
• Vikela Isitoreji: Ukuphindaphinda phakathi kwabathengi kuqinisekisa ukuthi idatha ihlala ikhona, noma ngabe i-node yehluleka.
• Ukutholakala kwendawo: I-Kafka ivumela amaqoqo ukuthi atshalwe ezindaweni ezahlukene ukuze kuthuthukiswe ukuqina nokufinyelela olwazini lomhlaba jikelele.
• Ukuhlanganiswa nezinhlaka Zedatha Enkulu: Njenge-Hadoop, Spark ne-Flink, ephindaphinda amathuba abo okusebenzisa.
• Ukuphatha okulula ngama-API namadeshibhodi: Ukuphatha kalula kanye nokuqapha.

Kuzo zonke lezi zizathu, i-Apache Kafka iwubuchwepheshe obaziswa kakhulu kwezezimoto zezimboni, ezentengiselwano zedijithali, amabhange, kanye namaphrojekthi wezokuxhumana lapho isekhondi ngalinye kubalwa futhi ukuqhubeka kwenkonzo kubalulekile.

Izinkampani eziphezulu ze-Apache Kafka kanye namacala okusebenzisa

Uhlu lwezinkampani eziye zamukela i-Kafka lubanzi ngempela futhi luhlanganisa izinkampani ezinkulu ezifana ne -Netflix, i-LinkedIn, i-Uber, i-PayPal, i-Cisco, i-Adidas, i-Oracle, i-Shopify, i-Spotify, i-Twitter, i-Trivago, i-Walmart, i-Microsoft Azure, i-Daumkakao , nezinye eziningi. Zonke zisebenzisa i-Kafka ukwakha izinhlelo zokuhlaziya idatha ezikwazi ukulinganiswa kanye nokucubungula, ukwenza ngcono izinhlelo zazo zokuthumela imiyalezo, ukuthuthukisa ulwazi lomsebenzisi, kanye nokwenza izinqubo ezibalulekile ngokuzenzakalela.

Empeleni, lokhu kuhumusha izimo zokusetshenziswa ezifana nalezi:

• Izaziso zesikhathi sangempela nezexwayiso kuzinhlelo zokusebenza zeselula newebhu.
• Ukuqapha ingqalasizinda ye-IoT namadivayisi.
• Ukuhlaziywa kokuziphatha kwekhasimende ukuze unikeze izincomo eziqondene nawe okwamanje.
• Ukuzenzakalela kwezimpendulo kanye nokuqapha izinqubo zezimboni.
• Ukuhlanganiswa kwamasistimu ahlukahlukene ezindaweni ezixubile noma ezinamafu amaningi.

Lokhu kusetshenziswa okujulile kubonisa inani lakho ekuphathweni kwedatha okuphumelelayo nokuthembekile kuyo yonke imikhakha eminingi.

Imiqondo ethuthukisiwe nezandiso ze-Apache Kafka

Kulabo abafuna ukuthola okuningi ku-Kafka, kunezici ezithuthukile ezikhulisa amandla ayo:

• I-Kafka Connect: Isixhumi esibonakalayo sokulayisha nokuthekelisa idatha kuya/kusuka kumasistimu ezinkampani zangaphandle ezifana nesizindalwazi, izinhlelo ze-ERP, amasistimu e-CRM, noma izinkundla zamafu. Ikuvumela ukuthi uxhume i-Kafka ecosystem kuyo yonke inhlangano yakho ngaphandle kwesidingo sokuthuthukiswa ngokwezifiso.
• Ukusakaza kwe-Kafka: Ilabhulali ye-Java eyisipesheli ekucubunguleni ukusakazwa kwedatha okuhle, ilungele ukudala izinhlelo zokusebenza ezihlaziya, eziguqulayo, nezisabela ezenzakalweni ngesikhathi sangempela.
• Izenzo zokuthengiselana: Kusukela inguqulo 0.11.0.0, i-Kafka isekela lezi zinhlobo zokusebenza, iqinisekisa ukucubungula "kanye ngqo", igwema ukuphindaphinda noma ukulahleka kwedatha.
• Ukusekela amaklayenti ngezilimi ezahlukene: Nakuba i-Kafka isebenzisa i-JVM, amaklayenti akhona e-Python, Go, .NET, C++, NodeJS, nezinye izilimi eziningi, okwenza kube lula ukuzisebenzisa kumaphrojekthi we-cross-platform.
• Ukuphathwa kwamafu nokusetshenziswa ku-Kubernetes: Kunamathuluzi nezinsizakalo eziphethwe ukwenza lula ukufaka, ukukala, nokuthuthukisa izinkundla ze-Kafka emafini.

Lezi zandiso kanye ne-ecosystem eqinile yezixhumi, imitapo yolwazi, namathuluzi okuqapha kuvumela i-Kafka ukuthi ijwayelane cishe nanoma yisiphi isidingo, kusukela ezindaweni zamabhizinisi kuya ekuqaleni kobuchwepheshe namaphrojekthi esayensi.

Izinselelo, ukuqapha kanye nezinqubo ezingcono kakhulu

Njenganoma yiluphi ubuchwepheshe obuthuthukisiwe, ukusebenzisa i-Kafka kuhilela nezinselele ezithile . Lokhu kufaka phakathi isidingo sokuhlela kahle izakhiwo, ukuphatha ukukhula kweqembu, kanye nokuqapha ukusebenza ukuze kutholakale izithiyo ezingaba khona.

Ukuze kube lula le misebenzi, kokubili amathuluzi omthombo ovulekile (njenge-LinkedIn's Burrow) kanye nezixazululo zezentengiselwano (njenge-Datadog) ayatholakala ukusiza ukuqapha impilo nokusebenza kwamaqoqo. Ngaphezu kwalokho, umphakathi we-Kafka unikeza imibhalo ephelele kanye nezinsiza eziningi zokuqeqesha, kusukela ezifundweni naseziqondisweni kuya ezifundweni ezikhethekile.

Ukusebenzisa imikhuba emihle kakhulu ekuhlukaniseni itimu, ukuhlela kahle ukugcinwa kwedatha kanye nokukopishwa kwayo, kanye nokuqapha izilinganiso ezibalulekile kuyizici ezibalulekile ukuqinisekisa ukusebenza kahle kwesistimu kanye nokuthembeka kwayo.

I-Kafka izimise njengenkundla eyisisekelo yokuphatha ukusakazwa kwedatha ngesikhathi sangempela ngenxa yezakhiwo zayo ezisabalalisiwe, ukuguquguquka, nokuqina. Ukwamukelwa kwayo emikhakheni eyahlukene kukhombisa ikhono layo lokubhekana nezinselele zedatha enkulu kanye noshintsho lwedijithali, ihlinzeka ngesixazululo esinokwehla, esithembekile, nesisebenza kahle kakhulu sokuphatha ulwazi olunyakazayo njalo.

I-athikili ehlobene:

I-Big Data Analytics: Ukuguqula Izinqumo