ag真人试玩

老司機獨愛Linux自動化運維神器saltstack,看他60分鐘的真言

身邊有很多運維工程師,做了幾年的運維自動化,但依然不能確定選擇哪個工自動化工作?還有,怎樣更優雅的實施運維自動化和避免事實當中的坑?
馬哥教育大咖講堂-175期特別邀請了一線資深工程師張強為我們帶來了一場《saltstack在自動化運維實踐》的分享,嘗試帶著他對工作問題的思考以及部署自動化工具的經驗給大家一些幫助和啟發。
————————————
作者介紹:
張強,馬哥教育Linux大咖講堂金牌講師,現就職于伙伴智慧運維工程師,負責主線業務平臺,3年Linux一線經驗,擅長shell腳本、自動化_發布、web應用等,現在關注自動化運維、分布式數據庫,虛擬化技術。


自動化工具比較

puppetyexushisikuangongjuzhongzuishenrurenxinde。jiukeyongcaozuo、mokuaiheyonghujiemianeryan,tashizuiquanmiande。puppetchengxianleshujuzhongxinxiediaodequanmao,jihuhangaimeiyigeyunxingxitong,weigedacaozuoxitongtigongleshenrudegongju。chushishezhibijiaojiandan,zhixuyaozaixuyaojiayiguanlidemeigexitongshanganzhuangzhufuwuqihekehuduandailiruanjian。minglingxingjiekou(cli)jiandanzhiguan,yunxutongguopuppetminglingxiazaiheanzhuangmokuai。ranhou,xuyaoduipeizhiwenjianjinxinggenggai,haorangmokuaishihesuoxuderenwu;yingjiedaozhilingdekehuduanyuzhufuwuqilianxishi,huigenggaipeizhiwenjian,huozhekehuduantongguolijichufagenggaipeizhiwenjiandetuisong(push)laijinxinggenggai。

ansibleguanzhudezhongdianshiliqiujingjianhekuaisu,erqiebuxuyaozaijiedianshanganzhuangdailiruanjian。yinci,ansibletongguosshzhixingsuoyougongneng。xuyaoguanlidejiedianbeitianjiadaoansiblepeizhihuanjing,sshshouquanmiyaobeifujiadaomeigejiedianshang,zheyuyunxingansibledeyonghuyouguan。yidanwanchenglezhebu,ansiblezhufuwuqikeyitongguosshyujiedianjinxingtongxin,zhixingsuoyoubiyaoderenwu。ansiblekeyishiyongparamiko(jiyussh2xieyidepythonshixian)huobiaozhunsshyongyutongxin,buguohuanyouyizhongjiasumoshi,yunxugengkuaisu、gengdaguimodetongxin。

ag真人试玩saltleisiansible,yinweitayeshijiyuclidegongju,caiyongletuisongfangfashixiankehuduantongxin。takeyitongguogithuotongguochengxubaoguanlixitonganzhuangdaozhufuwuqihekehuduanshang。kehuduanhuixiangzhufuwuqitichuqingqiu,qingqiuzaizhufuwuqishangdedaojieshouhou,jiukeyikongzhigaikehuduanle。saltkeyitongguoputongdesshyukehuduanjinxingtongxin,danruguoshiyongmingweiminiondekehuduandailiruanjian,keyidadazengqiangkekuozhanxing。ciwai,salthanyouyigeyibuwenjianfuwuqi,keyiweikehuduanjiakuaiwenjianfuwusudu,zhewanquanshisaltzhuzhonggaokuozhanxingdeyigetixian。yuansibleyiyang,nikeyizhijietongguocli,xiangkehuduanfachumingling,biruqidongfuwuhuoanzhuangchengxubao;niyekeyishiyongmingweistatedeyamlpeizhiwenjian,chulibijiaofuzaderenwu。huanyou“pillar”,zhexieshifangzaijizhongdifangdeshujuji,yamlpeizhiwenjiankeyizaiyunxingqijianfangwentamen。

zongjie:gerenguandianpuppetzuidaquedianjiushimorenqingkuangxiaagentmeige30fenzhongxiangmastertongbuzhuangtai,masterzhudongtuisonggongnengbijiaoboruo(2.7banben),ansiblejiyusshfuwuzhixing,ruguofuwuqiguoduobujianyishiyong,tashishiyonglunxundefangshi。saltjiyuxiaoxiduilie。xingnengxiangdanghao,shihedaliangshengchanhuanjing。

SaltStack簡介與特性

saltstack shiyizhongjiyu c/s jiagoudefuwuqijichujiagoujizhonghuaguanlipingtai,guanliduanchengwei master,kehuduanchengwei minion。saltstack jubeipeizhiguanli、yuanchengzhixing、jiankongdenggongneng,yibankeyilijieweishijianhuabande puppet hejiaqiangbande func。saltstack benshenshijiyu python yuyankaifashixian,jieheleqingliangjidexiaoxiduilieruanjian zeromq yu python disanfangmokuai(pyzmq、pycrypto、pyjinjia2、python-msgpack he pyyaml deng)goujian。

tongguobushu saltstack huanjing,yunweirenyuankeyizaichengqianshangwantaifuwuqishangzuodaopiliangzhixingmingling,genjubutongdeyewutexingjinxingpeizhijizhonghuaguanli、fenfawenjian、caijixitongshujujiruanjianbaodeanzhuangyuguanlideng。

SaltStack 具有以下特性:

1、bushujiandan、fangbian;

2、zhichidabufenunix/linuxjiwindowshuanjing;

3、zhucongjizhonghuaguanli;

4、peizhijiandan、gongnengqiangda、kuozhanxingqiang;

ag真人试玩5、zhukongduan(master)hebeikongduan(minion)jiyuzhengshurenzheng,anquankekao。

6、zhichiapijizidingyimokuai,ketongguopythonqingsongkuozhan。

SaltStack 的工作原理

saltstack caiyong c/s jiegoulaiduiyunhuanjingneidefuwuqicaozuoguanlijipeizhiguanli。weilegenghaodelijietadegongzuofangshijiguanlimoxing,jiangtongguotuxingfangshiduiqiyuanlijinxingchanshu。

ag真人试玩saltstack kehuduan(minion)zaiqidongshi,huizidongshengchengyitaomiyao,baohansiyaohegongyao。zhihoujianggongyaofasonggeifuwuqiduan,fuwuqiduanyanzhengbingjieshougongyao,yicilaijianlikekaoqiejiamidetongxinlianjie。tongshitongguoxiaoxiduilie zeromq zaikehuduanyufuwuduanzhijianjianlixiaoxifabulianjie。jutitongxinyuanlitu,rutu 1 suoshi,minglingzhixingrutu 2 suoshi:

老司機獨愛Linux自動化運維神器saltstack,看他60分鐘的真言

zhuanyeshuyushuoming:

ag真人试玩minion shi saltstack xuyaoguanlidekehuduananzhuangzujian,huizhudongqulianjie master duan,bingcong master duandedaoziyuanzhuangtaixinxi,tongbuziyuanguanlixinxi。

master zuoweikongzhizhongxinyunxingzaizhujifuwuqishang,fuze salt minglingyunxingheziyuanzhuangtaideguanli。

ag真人试玩zeromq shiyikuankaiyuandexiaoxiduilieruanjian,yongyuzai minion duanyu master duanjianlixitongtongxinqiaoliang。

daemon shiyunxingyumeiyigechengyuanneideshouhujincheng,chengdanzhufabuxiaoxijitongxinduankoujiantingdegongneng。

老司機獨愛Linux自動化運維神器saltstack,看他60分鐘的真言

yuanlitushuoming:

ag真人试玩minion shi saltstack xuyaoguanlidekehuduananzhuangzujian,huizhudongqulianjie master duan,bingcong master duandedaoziyuanzhuangtaixinxi,tongbuziyuanguanlixinxi。

master zuoweikongzhizhongxinyunxingzaizhujifuwuqishang,fuze salt minglingyunxingheziyuanzhuangtaideguanli。

master shangzhixingmoutiaozhilingtongguoduiliexiafadaogege minions quzhixing,bingfanhuijieguo。

SaltStack 的架構設計

weilerangdajiagenghaodelijie saltstack jizhonghuaguanlifangmiandeyoushi,yinci,genjuxiangmudeshijiqingkuanghuizhilebushujiagoutu,bingzaiwenzhongduijiagoutujinxinglexiangxishuoming。rutu 3 suoshi:

老司機獨愛Linux自動化運維神器saltstack,看他60分鐘的真言

shuoming:

ag真人试玩saltstack desuoyoubeiguanlikehuduanjiedian(rutu 3 suoshi db he web),doushitongguomiyaojinxingjiamitongxin,shiyongduankouwei 4506。kehuduanyufuwuqiduandeneirongchuanshu,shitongguoxiaoxiduiliewancheng,shiyongduankouwei 4505。master keyifasongrenhezhilingrang minion zhixing,salt youhenduokezhixingmokuai,birushuo cmd mokuai,zaianzhuang minion deshihouyijingzidaile,tamentongchangweiyunide python kuzhong,locate salt | grep /usr/?keyikandao salt zidaidesuoyoudongxi。

weilegenghaodelijiejiagouyongyi,yixiajiangzhanshizhuyaodeminglingfabuguocheng:

ag真人试玩saltstack de master yu minion zhijiantongguo zeromq jinxingxiaoxichuandi,shiyongle zeromq defabudingyuemoshi,lianjiefangshibaokuo tcp he ipc。

salt mingling,jiang cmd.run ls minglingcong salt.client.localclient.cmd_cli fabudao master,huoquyige jodid,genju jobid huoquminglingzhixingjieguo。

master jieshoudaominglinghou,jiangyaozhixingdeminglingfasonggeikehuduan minion。

ag真人试玩minion congxiaoxizongxianshangjieshoudaoyaochulidemingling,jiaogei minion._handle_aes chuli。

minion._handle_aes faqiyigebendixianchengdiaoyong cmdmod zhixing ls mingling。xianchengzhixingwan ls hou,diaoyong minion._return_pub fangfa,jiangzhixingjieguotongguoxiaoxizongxianfanhuigei master。

ag真人试玩master jieshoudaokehuduanfanhuidejieguo,diaoyong master.handle_aes fangfajiangjieguoxiedewenjianzhong。

salt.client.localclient.cmd_cli tongguolunxunhuoqu job zhixingjieguo,jiangjieguoshuchudaozhongduan。

SaltStack 的安裝與配置

dui saltstack youleyigechubudelejiezhihou,tongguoshijianlicaozuojinyibulejie saltstack。

yi、anzhuangsalt

saltxuyaoepelyuanzhichi,suoyouanzhuangqianxuyaoxiananzhuangepelyuanbao。

ag真人试玩1、salt-master

ag真人试玩# yum -y install salt-master

2、salt-minion

# yum -y install salt-minion

er、peizhisalt

1、master(/etc/salt/master)

ag真人试玩# saltyunxingdeyonghu,yingxiangdaosaltdezhixingquanxian

user: root

#saltdeyunxingxiancheng,kaidexianchengyueduoyibanchulidesuduyuekuai,danyibanbuyaochaoguocpudegeshu

worker_threads: 10

ag真人试玩# masterdeguanliduankou

publish_port : 4505

ag真人试玩# mastergenminiondetongxunduankou,yongyuwenjianfuwu,renzheng,jieshoufanhuijieguodeng

ret_port : 4506

# ruguozhegemasteryunxingdesalt-syndiclianjiedaoleyigegenggaocengjidemaster,neimezhegecanshuxuyaopeizhichenglianjiedaodezhegegaocengjimasterdejiantingduankou

syndic_master_port : 4506

# zhidingpidwenjianweizhi

pidfile: /var/run/salt-master.pid

# saltstack keyikongzhidewenjianxitongdekaishiweizhi

root_dir: /

# rizhiwenjiandizhi

log_file: /var/log/salt_master.log

# fenzushezhi

ag真人试玩nodegroups:

group_all: ‘*’

ag真人试玩# salt statezhixingshihoudegenmulu

ag真人试玩file_roots:

base:

– /etc/salt/

# shezhipillar degenmulu

ag真人试玩pillar_roots:

base:

– /etc/pillar

2、peizhiminion(/etc/salt/minion)

ag真人试玩master: mail? #zhekuaidemailzhideshizai/etc/hostswenjianzhongsuodingyidezhujiming

id: node1

3、qidongsalt

ag真人试玩service salt-master start

ag真人试玩service salt-minion start

# saltstack shishiyongpython2deyuyanbianxie,duipython3dejianrongxingbuhao,qingshiyongpython2dehuanjing

4、renzhengminglingjieshao

salt-key #zhengshuguanli

# salt-key –l ? ? ? #chakansuoyouminion-key

# salt-key –a ? ? ?#jieshoumougeminion-key

ag真人试玩# salt-key –a ? ? ?#jieshousuoyouminion-key

ag真人试玩# salt-key –d ? ? ? #shanchumougeminion-key

# salt-key –d ? ? ? #shanchusuoyouminion-key

老司機獨愛Linux自動化運維神器saltstack,看他60分鐘的真言

老司機獨愛Linux自動化運維神器saltstack,看他60分鐘的真言

老司機獨愛Linux自動化運維神器saltstack,看他60分鐘的真言

5、saltminglingjieshao

ag真人试玩minglinggeshi:salt [options] [arguments]

ag真人试玩li:salt \* cmd.run ‘uptime’

老司機獨愛Linux自動化運維神器saltstack,看他60分鐘的真言

SaltStack minion匹配方式

1、 glob(saltmorendetargetleixing,shiyongshelldetongpeifulaizhidingyigehuoduogeminion id)

# salt \* test.ping huo salt ‘*’ test.ping

2、pcrejianrongzhengzebiaodashi

ag真人试玩# salt –e ‘^[m|m]in.[e|o|u]n$’ test.ping

3、subnet(tongguozhidingyigeipv4dizhihuoyigecidrdeipv4ziwang)

# salt –s 192.168.0.42 test.ping

# salt –s 192.168.0.0/16 test.ping

ag真人试玩4、grains(saltkeyitongguocaozuoxitong、cpujiagoujizidingyixinxidengjiqitezhengjinxingtarget minion)

# salt –g ‘os:ubuntu’ test.ping

ag真人试玩# salt –g ‘os_family:debian’ test.ping

5、pillar(saltzhichitongguopillarshujujinxingpipei)

# salt –i ‘my_val:my_val’ test.ping

6、hunhe(compound)

ag真人试玩# salt –c ‘web* or g@os:arch’ test.ping

7、jiedianzu(nodegroup)

jiedianzuxuyaoshixiandingyi,peizhifangfaruxia:

# vim /etc/salt/master

ag真人试玩nodegroups:

ag真人试玩node: ‘l@node1,node2’

# salt -n node test.ping

SaltStack常用模塊

1、statusmokuai(chakanxitongxinxi)

老司機獨愛Linux自動化運維神器saltstack,看他60分鐘的真言

老司機獨愛Linux自動化運維神器saltstack,看他60分鐘的真言

# salt “*” status.diskstats? ? #chakancipanxinxi

ag真人试玩# salt “*” status.meminfo? ? ? #chakanneicunxinxi

ag真人试玩# salt “*” status.w? ? ? ? ? ? #wminglingfanhuixinxi

2、chakansuoyoumoduleliebiao

老司機獨愛Linux自動化運維神器saltstack,看他60分鐘的真言

3、chakanzhidingmoduledesuoyoufunctionfangfa

老司機獨愛Linux自動化運維神器saltstack,看他60分鐘的真言

4、chakanzhidingmoduleyongfa

老司機獨愛Linux自動化運維神器saltstack,看他60分鐘的真言

5、jutimokuaideshiyong(lizi)

老司機獨愛Linux自動化運維神器saltstack,看他60分鐘的真言

tongshidaozhidingjiqichakan

老司機獨愛Linux自動化運維神器saltstack,看他60分鐘的真言

ag真人试玩cmd.runmokuaideshiyong

老司機獨愛Linux自動化運維神器saltstack,看他60分鐘的真言

老司機獨愛Linux自動化運維神器saltstack,看他60分鐘的真言

Grains

Static bits of information that a minion collects about the system when the minion first starts.

the grains interface is made available to salt modules and components so that the right salt minion commands are automatically available on the right systems.

以上是官方的解釋,大致意思是說grains是minion第一次啟動的時候采集的靜態數據,可以用在salt的模塊和其他組件中。例如,當os_family的Grain數據為Centos時,則會使用yum工具組件來進行軟件包管理。Grains會在Minion進程啟動時加載,并緩存在內存中。這樣salt-minion進程就無須每次操作都重新檢索系統來獲取Grain,極大的提高了Minion的性能。

1、womenzhelijiandanzuoyigeshuchuceshi,keyikandaominionjiediandeyixiexinxiruxia:

老司機獨愛Linux自動化運維神器saltstack,看他60分鐘的真言

chakanjutimeiyixiangxinxi

老司機獨愛Linux自動化運維神器saltstack,看他60分鐘的真言

2、yingyongchangjing:

grainsdetexing–meiciqidonghuibao、jingtaijuedingletameiyoupillarlinghuo,yaozhidaopillarshisuishikebiande,zhiyaozaimasterduanxiugaileneiyibandouhuilikeshengxiaode。suoyigrainsgengshihezuoyixiejingtaideshuxingzhidecaiji,lirushebeidejiaose(role),cipangeshu(disk_num),caozuoxitongbanbendengzhurucileifeichanggudingdeshuxing。jiandanzongjieqilaigrainsdeyongturuxia:

(1),grainskeyizaistatexitongyingyongzhong,yonghupeizhiguanlimokuai。

(2),grainskeyizaitargetzhongshiyong,yonglaipipeiminion,biruos,yong-g。

(3),grainskeyiyongyuxinxichaxun,grainsbaocunzhushoujidaodekehuduandexinxi。

neimewomenjiukeyidedaoyigedazhidepanduan,ruguonixiangdingyideshuxingzhishijingchangbianhuade,neiqingcaiyongpillar,ruguoshihenguding、buyibiandeneiqingyonggrains。

3、grainsyouxianji

grainskeyibaochizaiminionduan、tongguomasterduanxiafadengduogefangshilaifenfa。danbutongdefangfayoubutongdeyouxianjide(youdidaogao):

ag真人试玩(1). /etc/salt/grains

(2) /etc/salt/minion

ag真人试玩(3)./srv/salt/_grains/? masterduan_grainsmuluxia

ag真人试玩youxianjishunxuyiciweicunzaizaiminionduan/etc/salt/minionpeizhiwenjianzhongdetongminggrainshuifugai/etc/salt/grainswenjianzhongdezhi,ertongguomasterduan_grainsmuluxiagrainswenjianxiafadezhikeyihuifugaiminionduandesuoyoutongmingzhi。bijiaoaokou,zongzhijide,tongguomasterxiafadegrainsyouxianjishizuigaodeke,/etc/salt/minioncizhi,/etc/salt/grainszuidi(core grainsbudadong,jiubutaolunle,zhegebi/etc/salt/grainshuandi)。

ag真人试玩4、grainsdexiafa

grainsdexiafadazhikeyifenweilianggesilu:

(1)zidingyide(_grains)keyitongguostate.highstate、saltutil.sync_grains、saltutil.sync_all dengfangfapiliangxiafa,qiejisuoyouzai_grainsmuluxiadesuoyouzidingyigrainszhidouhuixiafadaominion,zheshixuedejiaoxun。

(2)gudingcunfangzaiminionduanpeizhiwenjianzhong,rugrains、minionwenjianzhong,keyitongguofile managerdefangfaqupiliangxiafa/etc/salt/grainsdengpeizhiwenjianshixiangrainsdepiliangxiafa,dangranleyetongguobiedefangshibazhegewenjianpiliangxiafaxiaqu,doushiokde。

duibi:

(1)tongguostate.highstate xiafadegrainshaochushiwuxuzhongqiminionjikeshengxiao,dantongguoxiafa/etc/salt/grainswenjianxiafadegrainszhizebixuzhongqiminionduanfuwucaikeyishengxiao。

ag真人试玩(2)zidingyide_grainsmeicizaihighstatediaoyongdeshihoujiuhuizidongxiafa、shuaxin,er/etc/salt/grainswenjiandezebuhui。

Pillar

zaidaduoshuchangjingzhong,pillardebiaoxianxingweihegrainyizhi,danyougehendadequbieshi:pillarzaimastershangjinxingdingyi,cunzaiyuyigejizhonghuadelujing。pillarshujushiyutedingminionguanliande,yejiushishuomeiyigeminiondouzhinengkandaozijideshuju,suoyipillarkeyiyonglaichuandiminganshuju(zaisaltdeshejizhong,pillarshiyongdulidejiamisession,yeshiweilebaozhengminganshujudeanquanxing)。

ag真人试玩pillarkeyiyongzaineixiedifang:

1、minganshuju

ag真人试玩lirussh key,jiamizhengshudeng,youyupillarshiyongdulidejiamisession,keyiquebaozhexieminganshujububeiqitaminionkandao。

2、bianliang

keyizaipillarzhongchulipingtaichayixing,biruzhenduibutongdecaozuoxitongshezhiruanjianbaodemingzi,ranhouzaistatezhongyinyong。

3、qitarenheshuju

keyizaipillarzhongtianjiarenhexuyaoyongdaodeshuju。birudingyiyonghuheuiddeduiyingguanxi,mniondejiaosedeng。

ag真人试玩4、yongzaitargettingzhong

ag真人试玩pillarkeyiyonglaixuanzeminion,shiyong-ixuanxiang。

dingyipillar:

ag真人试玩masterpeizhiwenjianzhongdingyi:

morenqingkuangxia,masterpeizhiwenjianzhongdesuoyoushujudoutianjiadaopillarzhong,qieduisuoyouminionkeyong。ruguoyaojinyongzheyimorenzhi,keyizaimasterpeizhiwenjianzhongtianjiaruxiashuju,zhongqifuwuhoushengxiao:

pillar_opts: false

ag真人试玩shiyongslswenjiandingyipillar

pillarshiyongyustatexiangsideslswenjian。pillarwenjianfangzaimasterpeizhiwenjianzhongpillar_rootsdingyidemuluxia。shiliruxia:

pillar_roots:

base:

– /srv/pillar

xiamianzheduandaimadingyilebasehuanjingxiadepillarwenjianbaocunzai/srv/pillar/muluxia。yustatexiangsi,pillaryeyoutop file,yeshiyongxiangtongdepipeifangshijiangshujuyingyongdaominionshang。shiliruxia:

# cat /srv/pillar/top.sls:

base:

‘*’:

– packages

# cat /srv/pillar/packages.sls:

{% if grains[‘os’] == ‘redhat’ %}

apache: httpd

git: git

ag真人试玩{% elif grains[‘os’] == ‘debian’ %}

apache: apache2

git: git-core

{% endif %}

ag真人试玩basehuanjingzhongsuoyoudeminiondoujuyoupackageszhongdingyideshuju。pillarcaiyongyufile serverxiangtongdewenjianyingshefangshi,zaibenlizhong,packagesyingshedaowenjian/srv/pillar/packages.sls。zhuyikeyyuvalueyaoyongmaohaojiakonggefenge,meiyoukonggedehuajiangjiexishibai。

ruhezhidaominionyongyouneixiepillarshuju?

zaimastershangxiugaipillarwenjianhou,xuyaoyongyixiaminglingshuaxinminionshangdeshuju:

老司機獨愛Linux自動化運維神器saltstack,看他60分鐘的真言

ag真人试玩shiyongpillarhuoquzidingyishuju:

老司機獨愛Linux自動化運維神器saltstack,看他60分鐘的真言

State

jianshu:sls(daibiaosalt statewenjian)shisalt statexitongdehexin。slsmiaoshulexitongdemubiaozhuangtai,yougeshijiandandeshujugoucheng。zhejingchangbeichengzuopeizhiguanli

top.slsshipeizhiguanliderukouwenjian,yiqiedoushicongzhelikaishi,zaimaster zhujishang,morencunfangzai/srv/salt/mulu.

top.sls morencong base biaoqiankaishijiexizhixing,xiayijishicaozuodemubiao,keyitongguozhengze,grainmokuai,huofenzuming,laijinxingpipei,zaixiayijishiyaozhixingdestatewenjian,bubaohuankuozhanming。

chuangjian/srv/salt/top.sls

老司機獨愛Linux自動化運維神器saltstack,看他60分鐘的真言

stateshizhan

老司機獨愛Linux自動化運維神器saltstack,看他60分鐘的真言

老司機獨愛Linux自動化運維神器saltstack,看他60分鐘的真言

老司機獨愛Linux自動化運維神器saltstack,看他60分鐘的真言

————廣告時間————

《馬哥Linux云計算及架構師》課程,由知名Linux布道師馬哥創立,經歷了8年的發展,聯合阿里巴巴、唯品會、大眾點評、騰訊、陸金所等大型互聯網一線公司的馬哥課程團隊的工程師進行深度定制開發,課程采用 Centos7.2系統教學,加入了大量實戰案例,授課案例均來自于一線的技術案例。

開課時間:11月06號

相關新聞

AG捕鱼下载 AG真人