I-Qwen3-Omni: Konke odinga ukukwazi mayelana nemodeli ye-omnimodal

Informatec Digital » Intelligence yokufakelwa » I-Qwen3-Omni: Konke odinga ukukwazi mayelana nemodeli ye-omnimodal

Ukufika kwe-Qwen3-Omni kushintshe isimo se-AI: imodeli eyodwa yomdabu ekwazi ukuqonda nokuphendula umbhalo, izithombe, umsindo, kanye nevidiyo , enezimpendulo ezisheshayo kokubili ezibhaliwe nezikhulunywayo. Asikhulumi "ngama-patches" amaningi, kodwa kunalokho isakhiwo esiklanyelwe ngokuyisisekelo sokuhlanganisa izindlela ezine- latency ephansi kanye nokulawula ukuziphatha okulungisiwe kahle.

Ngesikhathi lapho cishe wonke umuntu ehlola ama-chatbot nabasizi, i-Qwen3-Omni ifika nesifiso esikhulu: isekela izilimi ezingu-119 ngombhalo, iqaphela inkulumo ngezilimi ezingu-19 futhi ikhuluma ngezilimi ezingu-10 , iqonda umsindo omude (kuze kube yimizuzu engu-30), futhi inamamaki okulinganisa ezivivinyweni eziningi. Ngaphezu kwalokho, indlela yayo yokuklama i-Thinker-Talker kanye ne-Mixture of Experts ihlose isivinini sempendulo kanye nekhwalithi yokucabanga ezimweni zangempela.

I-athikili ehlobene:

I-GPT-5: Konke mayelana noguquko olukhulu olulandelayo ku-Artificial Intelligence

Yini i-Qwen3-Omni futhi inikeza ini?

I-Qwen3-Omni umndeni wamamodeli ezilimi eziningi ayisisekelo, asebenzisa yonke indlela, kanye nasebenzisa izilimi eziningi kusukela ekuqaleni kuze kube sekupheleni aklanyelwe ukucubungula umbhalo, izithombe, umsindo, kanye nevidiyo, ngokukhipha kokubili umbhalo kanye nenkulumo yemvelo. Isihluthulelo asigcini nje ngokuhlukahluka kokufakwayo kanye nemiphumela kodwa futhi nokusebenza kwayo kokusakaza ngokuphendula okuguquguqukayo kwengxoxo kanye nekhono lokuphendula ngokushesha.

Ithimba lilethe ukuthuthukiswa okuningana kwezakhiwo kokusebenza kahle kanye nokusebenza kahle: ukuqeqeshwa kwangaphambi kwesikhathi "kombhalo kuqala" kuhlanganiswe nokuqeqeshwa okuxubile kwezindlela eziningi, kanye nomklamo one-Mixture of Experts (MoE) egcina ukusebenza kwayo embhalweni nasesithombeni ngenkathi ithuthukisa umsindo nevidiyo. Ngalezi zithuthukisi, imodeli ifinyelela i-SOTA kuma-benchmark angu-22 kwangu-36 omsindo/amavidiyo kanye ne-SOTA yomthombo ovulekile kuma-32 kwangu-36, nemiphumela efana ne-Gemini 2.5 Pro ku-ASR, ukuqonda umsindo, kanye nengxoxo yenkulumo.

Amakhono abalulekile nezindlela

I-Qwen3-Omni isilungele ukusetshenziswa kwezilimi zangempela zomsindo, umbono, kanye nezinhlelo zokusebenza ezilalelwayo, ngokusekelwa okukhulu kwezilimi eziningi: izilimi zombhalo ezingu-119, izilimi zokufaka ngezwi ezingu-19, kanye nezilimi zokukhipha izwi ezingu-10 . Izilimi zokufaka ngezwi zifaka phakathi isiNgisi, isiShayina, isiKorea, isiJapane, isiJalimane, isiRashiya, isiNtaliyane, isiFulentshi, iSpanishi, isiPutukezi, isiMalay, isiDashi, isi-Indonesian, isiTurkey, isiVietnamese, isiCantonese, isi-Arabhu, nesi-Urdu; kanti izilimi zokukhipha zifaka phakathi isiNgisi, isiShayina, isiFulentshi, isiJalimane, isiRashiya, isiNtaliyane, iSpanishi, isiPutukezi, isiJapane, nesiKorea, phakathi kwezinye.

Iqoqo lezincwadi zokupheka ezisemthethweni libonisa ububanzi bokusetshenziswa kwazo. Kumsindo, libonisa ukuqashelwa kwenkulumo okuhlukahlukene nokulalelwayo okude (i-ASR) , ukuhumusha kwenkulumo-kuya-kumbhalo kanye nokuhunyushwa kwenkulumo-kuya-kunkulumo, ukuhlaziywa komculo (isitayela, isigqi, izinhlobo), incazelo yomphumela womsindo, kanye nokubhalwa kwamagama-ncazo kwanoma yimuphi umsindo . Liphinde lisekele ukuhlaziywa okuxubile kwamathrekhi anenkulumo, umculo, kanye nemisindo ye-ambient.

Embonweni, inikeza i-OCR "enzima" yezithombe eziyinkimbinkimbi, ukutholwa kwezinto kanye nokusekelwa , i-QA yesithombe, ukuxazulula izibalo zesithombe (lapho imodeli Yokucabanga ikhanya khona), incazelo yevidiyo, ukuzulazula okusekelwe kuvidiyo yomuntu wokuqala, kanye nokuhlaziywa kokuguquka kwesigcawu . Ezimweni ezinomsindo nokubukwayo, ibonisa i-QA yomsindo nevidiyo ngokuvumelanisa isikhathi, ukusebenzisana okuqondiswayo nokufakwayo kwe-AV , kanye nezingxoxo nokuziphatha komsizi.

Njenge-ejenti, ivelele ngekhono layo lokusebenza ukushaya ucingo kusuka ku-audio , okuvula ukuhamba komsebenzi wezwi okusebenzisa amathuluzi, futhi emisebenzini esuselwe kukho kukhona i- Omni-Captioner yokubhala amagama angezansi ngemininingwane emihle, okubonisa ukujwayelekile kwalowo oyisisekelo.

I-Thinker-Talker Architecture ne-Design ene-MoE

Esinye sezizathu ezibalulekile zokuhlukanisa ukuhlukaniswa kwemithwalo yemfanelo: uMcabangi ukhiqiza umbhalo (onokuhlukahluka okuhlanganisa ukucabanga okucacile kochungechunge lokucabanga), kanti uMkhulumi ukhiqiza umsindo wesikhathi sangempela . Lokhu kuhlanganiswa kuvumela ingxoxo yezwi yemvelo ngenkathi uhlelo lugcina izinga eliphezulu lokuqonda umbhalo nokuhlela.

Isizindalwazi se-MoE sisabalalisa umthwalo womsebenzi phakathi kochwepheshe futhi sithembele ekuqeqeshweni kwangaphambi kwe-AuT ukuze kube nokumelwa okujwayelekile okunamandla. Ngaphezu kwalokho, ukusetshenziswa kokufaka ikhodi yamakhodi amaningi esiteshini somsindo kunciphisa ukubambezeleka okuncane, okubalulekile kumakholi noma kubasizi lapho yonke ingxenye yekhulu yesekhondi ibalwa khona.

Ukusebenza namabhentshimakhi: umbhalo, umbono, umsindo nokulalelwayo nokubonwayo

I-Qwen3-Omni igcina umbhalo osezingeni eliphezulu kanye nokusebenza kwesithombe ngaphandle kokwehla uma kuqhathaniswa namamodeli e-Qwen anobukhulu obufanayo agxile kwimodi eyodwa, kuyilapho ekusebenzeni komsindo nokubuka okulalelwayo kusetha ijubane ezivivinyweni eziningi . Ebhethrini lamabhentshimakhi angu-36 omsindo nokubuka okulalelwayo, ifinyelela i-SOTA yomthombo ovulekile ku-32 kanye ne-SOTA iyonke ku-22, idlula i- Gemini 2.5 Pro kanye ne-GPT-4o ngamaphuzu amaningana.

Ezinye izigaba ezibalulekile embhalweni: ku -AIME25 i-Flash-Instruct variant scores cishe engu-65,9; ku- ZebraLogic i-Instruct ifinyelela ku-90, kanti ku- MultiPL-E ifinyelela izibalo zokuncintisana ngokumelene ne-GPT-4o. Emisebenzini yokuvumelanisa efana ne-IFEval kanye ne-WritingBench, amamodeli e-Instruct kanye ne-Thinking abonisa amaphuzu aphezulu futhi ahambisanayo.

Kumsindo, imiphumela ye-ASR yesiShayina nesiNgisi mihle kakhulu: ku -WenetSpeech naku -LibriSpeech, kunciphisa kakhulu izinga lamaphutha egama, ngezibalo eziseduze no-1,22/2,48 ku-LibriSpeech clean/other, kanye namasethi afana ne -FLEURS (izilimi eziningi), inikeza amazinga aphansi kakhulu. Ku-VoiceBench, izibalo ezifana ne -AlpacaEval, i-CommonEval, ne-WildVoice zibeka i-Qwen3-Omni ngokulinganayo nezinhlelo zokubhekisela ezivaliwe, futhi idlula kakhulu ekucabangeni komsindo ku -MMAU v05.15.25.

Kuzinhlelo zokusebenza ezilalelwayo nezibonwayo, i-metric evame ukucashunwa kakhulu yi -WorldSense (cishe u-54,1 ), idlula i-Gemini-2.5-Flash. Ngaphezu kwalokho, kuma-suites afana ne -DailyOmni ne -VideoHolmes, i-Thinking variant ithola ukuthuthuka kunezinhlelo zokusebenza ze-SOTA zomthombo ovulekile zangaphambilini. Embonweni omsulwa, idlula kahle ku- MMMU, MathVista, MathVision , kanye nokuqonda amadokhumenti (AI2D, ChartQA), ngamaphuzu amahle kakhulu ekubaleni ( CountBench) kanye nokuqonda ividiyo (Video-MME, MLVU).

Ukukhiqizwa kwezwi okungadingi sithombe esiphelele nakho kwalinganiswa: uma kuqhathaniswa nemindeni efana ne-CosyVoice kanye ne-Seed-TTS, i-Qwen3-Omni ikhombisa ukuvumelana okungcono kokuqukethwe ezilimini eziningi kanye nokufana kwezikhulumi eziningi . Esigabeni sezilimi eziningi, amathebula athi "Ukuhambisana Kokuqukethwe" kanye "Nokufana Kwezikhulumi" abonisa ukuthi i-Qwen3-Omni 30B-A3B incintisana kakhulu ngesiShayina nesiNgisi, futhi iqinile ngesiJalimane, isiNtaliyane, isiPutukezi, isiSpanishi, isiJapane, isiKorea, isiFulentshi, kanye nesiRashiya. Ku -TTS yezilimi eziningi , ifinyelela i-WER/ukuhambisana okungcono phakathi kwamabili amaningana (isb., zh→en, ja→en, ko→zh) uma kuqhathaniswa ne-CosyVoice 2/3.

Amamodeli atholakalayo nokuthi ngalinye lisetshenziselwa ini

Umugqa we-Qwen3-Omni uhlanganisa izingcezu ezintathu eziyinhloko, ngayinye eyenzelwe ukusetshenziswa okuthile: Instruct , Thinking , kanye ne-Captioner . Zonke zivela enkabeni efanayo kodwa zinamakhono ahlukene asebenzayo noma ahlelwe kahle emisebenzini ethile.

I-Qwen3-Omni-30B-A3B- I-Instruct iqukethe i-Thinker and Talker, yamukela umsindo, ividiyo, nombhalo , futhi ibuyisela umbhalo nomsindo. Kuyisinqumo esifanele uma ufuna ukusebenzisana okugcwele nemiphumela ekhulunywayo ngesikhathi sangempela, futhi kunconywa kuma -demo ezwi noma evidiyo .

I-Qwen3-Omni-30B-A3B- Ukucabanga kugxila ku-Thinker ngokucabanga ngochungechunge , kusekela umsindo, ividiyo, kanye nombhalo ngokuphuma kombhalo. Kuwusizo ekuhlaziyeni okujulile, ekuxazululeni izinkinga eziyinkimbinkimbi, izibalo ezibonakalayo, noma emisebenzini lapho ungadingi khona ukuphuma kwezwi kodwa udinga ukucabanga okuhlelekile okungcono kakhulu.

I-Qwen3-Omni-30B-A3B- I-Captioner iyi-derivative ehlungiwe yomsindo ophansi kakhulu, onezihlokwana eziphansi . Ingumthombo ovulekile, ihlanganisa ububanzi bomsindo ngemininingwane eminingi, futhi igcwalisa igebe lomlando ohlelweni lwe-open-source: amagama-ncazo athembekile nacebile omsindo ojwayelekile.

Ukubambezeleka, isikhathi sangempela nokulawula ukuziphatha

Uhlelo lulungiselelwe ukusebenzisana okusheshayo, ngezikhathi zokuphendula ezingaba ngu-211 ms zomsindo kanye no-507 ms womsindo-ividiyo . Ngaphezu kokusakaza, kugxilwe ekuphenduleni okungokwemvelo kwengxoxo kanye nokulethwa kwezwi okuzinzile, kusizwa izindima ezicacile ze -Thinker (umbhalo) kanye ne-Talker (izwi).

Ukuze ulungise kahle, ungenza ngokwezifiso isitayela ngezixwayiso zesistimu . Ezimweni ze-AV lapho umsindo wevidiyo usetshenziselwa ukubhekisela khona, ithimba liphakamisa isixwayiso sesistimu esigcina ukucabanga kwe-Thinker ngenkathi sinikeza umbhalo ofundeka kalula nongaxoxwayo, okwenza kube lula nge- Talker ukukhuluma kahle . Kunconywa futhi ukugcina ipharamitha ethi `use_audio_in_video` ihambisana kuyo yonke ingxoxo ephindaphindwayo.

Ekuhloleni, kuneziqondiso ezithile: ungabeki i-system prompt , landela ifomethi ye-ChatML yesilinganiso ngasinye futhi, uma kungekho prompt, sebenzisa okulandelayo ngokuzenzakalelayo: i-Chinese ASR (“请将这段中文语音转换为纯文本。”), ezinye izilimi ze-ASR ("Bhala umsindo ube umbhalo."), i-S2TT (“Lalela inkulumo enikeziwe ye-<source_language>…”), kanye namagama engoma (“ Bhala amagama engoma” … ngaphandle kwezimpawu zokubhala, imigqa ehlukaniswe ngamakhefu”).

Ukuthunyelwa, izidingo namathuluzi

Ukuze uthole ulwazi oluphelele lwendawo, ithimba lincoma ama-Hugging Face Transformers kanye nokubukeza izigaba zobunjiniyela besofthiwe , kodwa qaphela: njengoba kuyisakhiwo se-MoE, singasebenza kancane nge-HF inference; ngezinhlelo zokusebenza zokukhiqiza noma ze-low-latency , bayeluleka ukusebenzisa i-vLLM noma i-DashScope API , futhi banikeze ngisho nesithombe se-Docker esifaka phakathi izindawo zazo zombili. Ikhodi yama-Transformers isivele ihlanganisiwe, kodwa iphakheji ye-PyPI ayikakhishwa futhi kumele ifakwe kusuka emthonjeni.

Banikeza izinsiza zokuphatha umsindo nesithombe/ividiyo (i-base64, ama-URL, okokufaka okuhlanganisiwe), futhi batusa i-FlashAttention 2 ene-Transformers ukunciphisa imemori ye-GPU noma nini lapho ilayisha ku- float16 noma ku-bfloat16 . Nge-vLLM, i-FlashAttn2 ifakiwe, futhi amapharamitha anjenge- limit_mm_per_prompt (i-pre-allocates GPU memory) kanye ne -max_num_seqs yokufana kuchazwe kabanzi ; ngaphezu kwalokho, ukwandisa i-tensor_parallel_size kwenza kube lula ukuphetha kwe-multi-GPU.

Kunezeluleko ezithile eziwusizo zokonga izinsiza: uma ungadingi umsindo, ungakhubaza i-Talker ngemva kokuqalisa, wonge cishe i-10 GB ye-VRAM. Futhi uma ufuna ukukhishwa kombhalo okusheshayo, sebenzisa i-`return_audio=False` ngesikhathi sokukhiqiza. Izidingo zememori eziyisisekelo ze-BF16 ene-FlashAttn2 nazo zinikeziwe: isibonelo, i-Instruct 30B-A3B isebenzisa cishe i-78,9 GB ngemizuzwana eyi-15 yevidiyo kanye ne-144,8 GB ngemizuzwana eyi-120; i-Thinking isebenzisa cishe i-68,7 GB kanye ne-131,7 GB, ngokulandelana.

Ukuze usethe i -demo yewebhu yendawo , batusa ukulungiselela indawo yakho ye-vLLM (noma indawo ye-Transformers ehamba kancane), ukuqinisekisa ukuthi ufake i-ffmpeg , nokusebenzisa izikripthi zabo. Banikeza izithombe ze-Docker ezilungele i-GPU “qwenllm/qwen3-omni” nge -NVIDIA Container Toolkit , i-port mapping (isb., i-host 8901 → container 80), kanye nenketho yokukhonza kusukela ku-0.0.0.0. Ungaphinda ufake noma ususe isitsha njengoba kudingeka.

Amademo, ama-API, ne-ecosystem

Uma ungafuni ukusebenzisa endaweni yangakini, ungazama ama-demo ku-Hugging Face Spaces kanye ne-ModelScope Studio , kanye nokuhlangenwe nakho kwe-Qwen3-Omni-Realtime, Instruct, Thinking, kanye ne-Captioner. I-Qwen Chat enokusakaza kwangempela nayo iyatholakala: mane ukhethe inketho yezwi/yekholi yevidiyo ku-interface.

Ukuze kuhlanganiswe kahle nokubambezeleka okuphansi, indlela enconywayo yi- DashScope API , enikeza ukusebenza okubikezelwa kakhulu. Ngaphezu kwalokho, umphakathi uxhumana ngeziteshi ezifana ne -Discord kanye ne-WeChat , futhi ushicilela izincwadi zokupheka ezinamalogi okusebenza angempela avumela abasebenzisi ukuthi bakhiqize imiphumela ngokushintsha izixwayiso noma amamodeli.

Imephu yomgwaqo kanye nokuthuthukiswa okuqhubekayo

Ithimba lisebenza kwezinye izici ezifana nokuqashelwa kwenkulumo yezikhulumi eziningi , i-OCR esetshenziswa kuvidiyo, ukuthuthukiswa kokufunda okubonakalayo okusebenzayo, kanye nokusebenza kwe-ejenti okucebile. Baphinde babonisa ukuthi ukwesekwa kokukhipha umsindo ku-vLLM yemodeli ye-Instruct kuzotholakala maduze, kuqedele umjikelezo wokuthunyelwa kwesikhathi sangempela kusukela kuleyo backend.

I-FAQ: Ukusekelwa kwesikhathi sokusebenza kanye nokulinganisa

Abanye abasebenzisi baphawule ngokuthi abakwazi ukusebenzisa i-Qwen3-Omni ngisho noma benama-suspect ajwayelekile nokuthi ababoni ama-quants ku-Hugging Face ; ngaphezu kwalokho, ifomethi yendabuko ye-16-bit icishe ibe ngu-70 GB, usayizi oyinkinga kumakhompyutha amancane. Iphrojekthi ngokwayo icacisa ukuthi ama-Transformers asevele ehlanganisiwe kodwa ngaphandle kwephakheji ye-PyPI , okumele ifakwe kusuka emthonjeni, nokuthi i-vLLM iyindlela ekhethwayo yokucabanga, yize ukwesekwa komsindo kwe-Instruct ku-vLLM kuzokhishwa maduze.

Ngokuphathelene nokulinganisa, azikho izibambi-ndawo ezibalwe ku-HF ze-Qwen3-Omni 30B-A3B, futhi kufanelekile ukukhumbula ukuthi i- MoE kanye nemvelo ye-multimodal kwenza kube nzima ukuhambisana ngokushesha nezikhathi zokusebenza ezifana ne-llama.cpp. Kulabo abadinga ukuhlola manje, isincomo esisemthethweni ukusebenzisa i-Docker + Transformers/vLLM kusuka emthonjeni noma ku- API , futhi uhlale uqaphile endaweni yokugcina ukuze uthole izicelo zokudonsa ukwesekwa kanye nokulinganisa kwesikhathi esizayo uma zitholakala.

Imikhuba emihle yokuhlola kanye nemiyalelo

Ukuze kukhiqizwe izinombolo, iziqondiso ezilandelayo zichaziwe: ama-benchmark amaningi asebenzisa i-greedy decoding ku-Instruct ngaphandle kwe-sampling, futhi ku-Thinking, amapharamitha ku- generation_config.json kumele ahlonishwe . Izinga lohlaka lwevidiyo liphinde lisethwe ku- fps=2 ngesikhathi sokuhlolwa, futhi kuboniswa ukuthi isixwayiso somsebenzisi kufanele silandele idatha ye-multimodal ngaphandle kokuthi isethi yedatha isho okunye.

Uma i-benchmark ingafaki i-prompt, i-prompt ezenzakalelayo ingasetshenziswa (i-Chinese ASR/other, i-S2TT, amagama engoma). Ngaphezu kwalokho, i-prompt yesistimu akufanele isethwe ngesikhathi sokuhlola ukuqinisekisa ukuthi imiphumela iyafana kuzo zonke izinhlelo kanye nokusebenza.

I-Qwen3-Omni izibeka njengeplatifomu yangempela ye-omnimodal, ene-latency ephansi, ukwesekwa okubanzi kwezilimi eziningi, amakhono omsindo namavidiyo asezingeni eliphezulu , kanye nendlela ecacile yokusabalalisa esebenzisa ama-Transformers, i-vLLM, kanye ne-Docker. Kulabo abafuna imodeli eyodwa ephatha umbhalo nezithombe ngaphandle kokulahlekelwa ukusebenza, futhi elalela, ekhuluma, futhi eqonda ividiyo , kuyisiphakamiso okunzima ukusinqoba namuhla.