KeeperoftheKeysRichlv: 3.4.4
snuff-workwith the odbc checks that you can use in zabbix.. (aka LLD etc), can they be executed on any agent or only on the zabbix master?
robin22Hi All, someone can help me please ? I want use a network map but i don't find the good syntax for show the rate IN and OUT for SNMP Template interface, for exemple i have try like this -> {{HOSTNAME}:ifOutOctets[Port #23]} but it's doest work :-(
finalbetaI think I know the answer, but I'm going to shoot anyway as I have no experience with it. Can I give a user more granular rights? I want a user to be able to ack triggers, disable/enable triggers, but I don't want him to be able to add/remove hosts or add templates.
finalbetaI think the answer is, "no way" :)
tuxickwhat's the deal with this $$ stuff in userparameters ?
Ticho_it's explained at https://www.zabbix.com/documentation/3.4/manual/config/items/userparameters
tuxickah ok, so it is consistent. seemed a bit random to me :)
tuxickanyway, trying to fix borg backup monitoring
zabbixbot[04ZBX-13112] 03Selecting box on dashboard's graph can cause redirect to graph page 14created by Larisa Grigorjeva14/larisa.grigorjeva (12https://support.zabbix.com/browse/ZBX-13112)
zabbixZBX-13112 Open Updated today -- Selecting box on dashboard's graph can cause redirect to graph page
CHYCIf I have multiple servers polling away, is there a way to get the history from one into another? The reason I want this over proxies is that we have multiple teams and want separate instances with superadmin access to each. With admin access there doesn't seem to be a way to limit the proxies you can attach a host to.
tuxickUserParameter=borg.all_original[*], grep -i "all archives" status/$1-status.txt | awk '$$4~/^GB/{printf "%.f", $$3*1000*1000*1000}; $$4~/^MB/{printf "%u", $$3*1000*1000}; $$4~/^kB/{printf "%u", $$3*1000}; $$4~/^B/{print $$3};'
tuxickthis is missing ^TB, but it made me wonder, is this really a good way to handle things?
tuxickin fact, how am i supposed to deal with terabytes at all? isn't that exceeding size of numeric/unsigned?
stixHi guys. Running Zabbix 3.4 on a mysql-cluster (galera) where zabbix connects to a VIP which is automatically moved between the mysql-servers. Sometimes we have experienced that we receive no alerts at all, after a VIP mode. Zabbix-server needs a restart. How can I best monitor the health of the alerter processes or alert manager?
tuxickshouldn't that be fixed at the source/cause?
tuxickwhy would it need a restart?
tuxicksounds to me like the vip switching screws up
stixtuxick: I don’t know. Happened this weekend. Our on call duty guy wondered why he hadn’t received any SMS’es for a week. Then he restarted the zabbix-server program and received 100+ SMS’es from the past week
stixtuxick: sounds to me like zabbix is unable to detect the switch and maybe reconnect
tuxickstix: it shouldn't even notice
stixtuxick: if it has a persistent connection to a database-server, then yes
tuxickBUT, can you check the system load on the zabbix server before/after that switch?
stixyes, there’s no load issues
tuxickok
tuxickso definitely no high load until restart?
tuxickbecause i've seen that happen after network hickups
stixpositive
tuxickok
tuxickstill, i've done network/IP failovers before, client shouldn't notice
glebtuxick: "Tera" is way below limits of 64 bit unsigned numbers
tuxickyes, but can zabbix/database handle that
glebtuxick: "Numeric (unsigned)" can
Floflobelhello, I install zabbix with postgres database but when I try to connect to my database it's return "Cannot connect to the database. System error occurred. Please contact Zabbix administrator.". Where I can view logs to see if it's a problem of access or right on the database?
zabbixHello Floflobel, and welcome to #zabbix. I'm your friendly house bot :) Please be patient when it takes a while for people to reply. See our channel guidelines on https://zabbix.org/wiki/Getting_help#IRC and have a nice stay!
glebtuxick: Look, 16 exabytes: https://support.zabbix.com/browse/ZBX-3388 :D
tuxickgleb: maybe that's got a M multiplication?
tuxickstix: i'd really first check your VIP failover
tuxickclient shouldn't notice
stixtuxick: but clients need to handle this, if the connection to the database-server is down, they should reconnect
tuxickstix: isn't that what HA/clustering is about? avoiding that situation?
tuxickbut i agree, client should be able to cope
glebtuxick: Conclusion is you can store integers up to 2^64 - 1 as Numeric (unsigned) in Zabbix
tuxickgleb: i can't conclude that from zbx-3388
glebtuxick: Multipliers are applied before writing into DB. Graphs just display what's in DB. If they display 16 EB, conclusion is...
tuxickreally?
tuxickhold on, there's multiplier and units
tuxickif unit is M, what is stored?
Andris_zbxFloflobel: could it be that your frontend version is newer than database (Zabbix schema) version ? If you start server it will upgrade database and frontend will start working.
tuxickMB
FloflobelAndris_zbx, thank you after restart and connect to my database with zabbix server it's works
glebtuxick: Units do not affect storing, only displaying.
stixtuxick: no not in this case. The point is to have automatic failover
tuxickgleb: ack :)
tuxickstix: yes, so client shouldn't even notice :)
tuxickbut i've seen crappy software failing on temporarily losing connection to database
tuxickthink it was java stuff
troffaskyshould a discovery rule OID have a "Latest Data" entry?
troffaskyi have a discovery rule with an OID listing the number of open snapshots but none of the item prototypes get created
kimbarbelhi, after upgrade zabbix to 3.4 we experienced big memry leaks - this leaks are related with zabbix pollers - we have got 400 pollers and its consuming of ram is still growing
kimbarbeldoes anybody have clue what could be a reason ?
troffaskyhow much does each poller use? about 60MB here [RES]
kimbarbelabbix 435 0.0 0.1 1979408 314504 ? S Nov26 0:23 zabbix_server: poller #380 [got 0 values in 0.000003 sec, idle 1 sec] zabbix 436 0.0 0.1 1982076 316536 ? S Nov26 0:23 zabbix_server: poller #381 [got 0 values in 0.000003 sec, idle 1 sec] zabbix 437 0.0 0.1 1962532 304420 ? S Nov26 0:21 zabbix_server: poller #382 [got 0 values in 0.000003 sec, idle 1 sec] zabbix 438 0.0 0.1 19726
kimbarbelfor example: zabbix 444 0.0 0.1 1984616 320312 ? S Nov26 0:22 zabbix_server: poller #389 [got 0 values in 0.000003 sec, idle 1 sec]
kimbarbelfrom yesterday
kimbarbelzabbix 444 0.0 0.1 2072312 403772 ? S Nov26 0:36 zabbix_server: poller #389 [got 0 values in 0.000019 sec, idle 1 sec]
kimbarbeltoday
kimbarbelover 200MB
kimbarbelat this time we have workaround and we restarting zabbix after it takes 100 GB of RAM
troffaskylol, that's some workaround
troffaskyi am pretty sure i am not seeing this problem with 3.4
troffasky80 pollers, ~60MB each
troffaskyzabbix uptime is 1 week
kimbarbelwe have got 400 pollers and every one ram usage is growing to over 200 MB
tuxicksounds leaky
kimbarbelexactly
kimbarbelhttps://uploadfiles.io/ljx97
kimbarbelhttps://ufile.io/ailuz
kimbarbelthis leaks started exactly after upgrade
Richlvkidn3ys, could you give an example on what is recursive there ?
Richlvkidn3ys, it seems like lld or dynamic indexes might be what you are looking for
Richlvkimbarbel, which exact 3.4 ?
kimbarbelyes
RichlvACTION blinks
troffaskyACTION rolls on floor
zabbixbot[04ZBX-13113] 03zabbix-agent depends libssl1.1 14created by Celio di Cavalcanti14/celiocavalcanti (12https://support.zabbix.com/browse/ZBX-13113)
zabbixZBX-13113 Open Updated today -- zabbix-agent depends libssl1.1
tuxicksounds like debianism
tuxickone of the stupidest error messages i ever saw
tuxickzabbix-agent : Depends: libssl1.1 (>= 1.1.0) but it is not installable
zabbixbot[04ZBX-13114] 03selectLastEvent on trigger.get returnes empty array instead of null 14created by Jacob Mansfield14/cyberjacob (12https://support.zabbix.com/browse/ZBX-13114)
zabbixZBX-13114 Open Updated today -- selectLastEvent on trigger.get returnes empty array instead of null
troffaskytuxick, do you have a suggestion for a better error message?
tuxickzabbix agent depends on libbsll > 1.1.0, but this cannot be installed for <insert proper reason>
tuxickother than the fact i believe 1.1 >= 1.1.0
troffaskyyou would see the reason if you attempted to install libssl1.1
Richlvtuxick, you are reading parts of it wrong
Richlvthe part in parenthesis is a clarification
Richlvso it says "this depends on libssl1.1"
Richlv"and by that i mean >= 1.1.0"
Richlvand yeah, reason is in another line in the output
troffaskybut whatever the reason is, it's not a zabbix issue
troffaskyi think "bugs and issues" tracker is just a duplicate of the forums now
vaizkiI just upgraded from 2.2 -> 3.4 and it took me half an hour.. impressive :O
tuxickRichlv: i don't conside it a zabbix problem, apt has been annoying me with such messages for years :)
tuxickit's just very unclear
troffaskyit's a matter of personal taste i suppose - i wouldn't want a recursive torrent of errors about dependencies and dependencies of dependencies when failing to install something
Richlvtroffasky, it has always had a tendency to be a bit abused
Richlv(the tracker, that is)
Richlvone just has to clean it up all the time
Richlvvaizki, congrats :)
troffaskyi think each so-called "bug" report should require a forum thread before being opened :-P
Richlvtroffasky, did you comment because of a specific issue, btw ?
Richlvtroffasky, nah, an irc discussion ;)
vaizkiRichlv, more like congrats for a smooth upgrade procedure!
Richlvvaizki, zabbix server devs for the win on that one
troffaskyRichlv, no, not a specific one - there are loads of "bug" reports that are just people asking for help
troffaskyi suppose being called "BUGS AND ISSUES" invites people to open, er, issues rather than just bugs
troffaskyso maybe it's working as intended
Ingus_https://support.zabbix.com/browse/ZBX/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel
Ingus_The project is for bug reports.
Ingus_This is NOT for feature requests, consultancy, support requests. Please, use Zabbix forums instead.
troffaskyi guess issues needs dropping from the name then
Ingus_And would that stop the users from posting just another Ineedfreehelpasap111 ZBX?
troffaskyonly one way to find out
Ingus_We had a discussion lately right about this that the address SUPPORT.zabbix.com is also misleading...
Ingus_But at the same time we have customer projects in the same JIRA.
troffaskyperhaps non-customers should have to jump through some hoops before opening bugs
ccookeIn an open source, free download world where the packages are hosted in many third-party locations, how do you reliably determine who is and is not a customer?
Ingus_troffasky, I am aftaid that is not possible in current JIRA setup.
troffaskyif the customer projects are in JIRA then presumably they have some information about the customer
Ingus_Indeed, customers do have their own logins created by Zabbix SIA and permissions to their own restricted projects.
Richlvhost boot time : 2009.07.02 11:28:07
Richlvpretty decent for a server
Ingus_That, however, does not prevent from posting in ZBX or ZBXNEXT public projects which is a guud and bad thing at the same time.
troffaskyi guess whoever does janitorial work on ZBX{,NEXT} would have the motivation to address this
Richlvi recall the rate of "non-bugs" was manageable
Richlvduplicates were more frequent :)
Richlvbut that's just the culture of the reporters
Richlvyou kinda have to work with them and educate them. later they'll come back with great reports
Ingus_Yeah, got to ask the janitors :D
Ingus_Duplicates are indeed problematic. And not always you can find the original issue even if there is one.
Ingus_Rich, thanks btw for pointing out those every now and then!
RichlvIngus_, you're very welcome - i'd love to do it more frequently, but time constraints kick in
Richlvto find dupes, one has to almost know the whole bug/feature request by heart. at least to recall that "something similar was there..."
Richlv...whole db
Ingus_Sure, that is all understandable. Same here he he
Ingus_yeah, it´s hard to find track of all issues. I know mine a bit but not all the db. Same for customer tickets.
RichlvIngus_, report triaging (bugs, feature requests) is an area where a crack team of community wizards could help a lot, if encouraged and nurtured
Ingus_Do you have any ideas how?
Ingus_(for the last part)
Ingus_Because we have been thinking about this.
Richlvi've had many ideas, but going over those might be easier in a meetup-environment. that might be one thing that could finally push me to organise some small recurring local meetup
Richlvwinter is better to start that, as it's dark outside so people have more interest in indoor things
troffaskyZabbix User Meeting
zabbixtroffasky: incorrect usage, ask for help using 'zabbix: help user'
troffaskycf. Mikrotik User Meeting
Richlvtroffasky, plain "user meeting" would be rather sad
Ingus_Drop me a line when you feel ready for such event and I´ll try to get things moving from Zabbix side.
zabbixbot[04ZBX-13115] 03Converting item prototype from zabiix agent to dependent item fails 14created by Sergey14/serg_k (12https://support.zabbix.com/browse/ZBX-13115)
zabbixZBX-13115 Open Updated today -- Converting item prototype from zabiix agent to dependent item fails
RichlvIngus_, cool, thanks. for the first one not much to get moving i believe, just have to kick off a recurring thing. will make sure to push myself for december
timelessdoes anyone have an android app for zabbix that works?
ftzdominowhat does it mean if my active queue items are high? do i need to start more runners on the agents for processing active checks?
volterYou can't have more than one.
volterHowever, the "queue" is no queue either.
volterIs that for certain hosts or all over the place?
ftzdominoit's all over the place.
smuckAny good hint how to debug my enviroment or what could be the reason that i see latest data in 30min latency. after restarting zabbix-server all are running fine for a while.
smucki think it started after we went 3.4.4
volterftzdomino: First take a look at how busy your server/proxy processes are!
volter!t internal
zabbixbotinternalmonitoring: Internal items can help to figure out how full caches are, how busy internal processes are etc. This can be used to determine how many processes are needed. Zabbix ships with default templates for internal monitoring. See http://blog.zabbix.com/monitoring-how-busy-zabbix-processes-are and https://www.zabbix.com/documentation/3.2/manual/config/items/itemtypes/internal
voltersmuck: OK, could be a bug then too.
volterOh, hang on, never mind what I said.
ftzdominovolter: that seems okay: https://jonsnell.s3.amazonaws.com/Custom_graphs_refreshed_every_30_sec._2017-11-29_12-02-25.png
ftzdominowould a bunch of very slow active checks cause issues?
volterYes
smuckam thinking about that could it be db syncers
smuck:32 /usr/sbin/zabbix_server: history syncer #9 [synced 5295 items in 68.996938 sec, syncing history]
ftzdominoshould i switch the slower ones to passive checks, then?
volterFor instance, yes.
Death_SynHello
zabbixHello Death_Syn, and welcome to #zabbix. I'm your friendly house bot :) Please be patient when it takes a while for people to reply. See our channel guidelines on https://zabbix.org/wiki/Getting_help#IRC and have a nice stay!
Death_SynI've done some googling, but found nothing confirming/denying so far... Does zabbix-agent require a writable file system to work? I've got some embedded devices I need to monitor, and to avoid wear on the flash, we have the file systems configured read-only. Will zabbix-agent work okay in this case?
Richlvit usually wants to place pidfile and logfile somewhere
smuckdeath_syn: no you can actual start agent on read-only enviroment. i did that on freenas years ago.
Richlvin recent versions log can be set to stdout, too
smuckif you can script you can do it.
smucklike startup script .
Richlvand pid can probably be disabled, if i recall crrectl
Richlvcorrectly.
Richlvso should be configurable to not do any writing
Death_Synokay
Death_Synthank you
_ikke_You could use tmpfs for things like /tmp and /var/run
Death_Synyeah, already do
_ikke_right
Death_Synokay, everything i see so far is encouraging. thank you
ftzdominoi've switched a bunch of items over to passive checks. thanks for the help, volter!
smuckvolter: thanks for reply. i increased db syncers so maybe that will do the work.
smuckdb is hitting nearly 1TB but so far performance does not seem to be issue.
_ikke_smuck: do you use partitioning?
smuck_ikke_: not at the moment but i have been thinking about mysql partitioning. so far i have tryed to avoid that.
smuckhistory_uint and trend is getting fat.
paulwasianyone successfully used a userparameter item key which takes an argument? I'm getting "invalid parameter" when I execute it. syntax used per google is UserParameter=test[*],/path/to/script $1
_ikke_smuck: yeah, and the problem is that the housekeeper can't keep up (at least for us)
paulwasihttps://www.zabbix.com/documentation/3.4/manual/config/items/userparameters toward the bottom has the example config
smuck_ikke_: indeed we are using "MaxHousekeeperDelete=600000" but had problems before after i moved DB and cleaned history_uint
smuck>abbix busy history syncer processes is hitting 100% all the time:)
Richlvsmuck, i tend to reduce history retention as much as possible and using reasonable (read - higher) intervals
Richlvthat is, i do that to delay the need for the partitioning until zabbix natively supports it :)
Richlvpaulwasi, how do you execute it ?
paulwasiok I just figured it out, when you run it with zabbix_Agentd -t you have to put the argument in []
Richlvoh
Richlvdon't tee-pee ;)
Richlvhttp://www.zabbixbook.com/2017/08/22/debugging-zabbix-userparameters/
Richlvbut you have to do "key[param, param param]" syntax everywhere
Richlvas in, _get, items etc
timelessis there a zabbix app for android that works better than Tabbix?
timelessACTION at least got that to work...
paulwasithank you Richlv I won't -t -p anymore. just seemed faster than setting up a host with an item and waiting for the config refresh and checking monitoring latest data
Richlvzabbix_get
Richlvas the article suggests :)
paulwasidoesn't that require [] and also a host with the item key assigned to it?
Richlv<Richlv> but you have to do "key[param, param param]" syntax everywhere
Richlvbut no host needed
Richlvzabbix_get just connects to the agent, zabbix server does not exist for it
paulwasionly problem with that is find / -name "*zabbix_get*" comes up blank
Richlvhow did you install zabbix ?
Richlvand on which system did you look for it ?
paulwasithis box is a proxy + agent box, installed from repo and yum install I believe
Richlvthen install zabbix get package
Richlvit is likely a separate one
paulwasizabbix_get [27112]: Check access restrictions in Zabbix agent configuration
paulwasiseems like more trouble than -t since you need to use all of the same syntax + connection details
paulwasiok it just didn't like localhost... hostname worked
smuckRichlv: thank you very much for help. not sure if i understood what did you mean... So lower housekeeping cycle and doing higher HousekeepingFrequency, currently "HousekeepingFrequency=1"
paulwasiRichlv ++
paulwasiRichlv++
zabbixKarma for Richlv is now 174
smucki have tryed to optimize intervals alot but the problem is that the enviroment is growing fast and events per second is pretty high even that i allways lower intervals and history .)
Richlvsmuck, i tend to keep less history and poll less frequently to have smaller db, which in turn results in a faster housekeeper
hydruidI'm having issues using the API in a Zabbix::Tiny script
hydruidTemplate import script: https://pastebin.com/QnWK2DPV
hydruidIf I run the script from the command line it works fine. ./import_script.pl (JSON-text)
hydruidif I run it from a bash script that cat's a file, which contains (JSON-text) it says "Error: Cannot read JSON: unexpected character."
Richlvhydruid, probably drops quoting and/or escaping
Richlvpassing such things on the cli is very error prone
Richlvmaybe stuff it in from a file or something
iskorptixhello boys
zabbixHello iskorptix, and welcome to #zabbix. I'm your friendly house bot :) Please be patient when it takes a while for people to reply. See our channel guidelines on https://zabbix.org/wiki/Getting_help#IRC and have a nice stay!
iskorptixI'm tracking source of one very odd issue
iskorptixthe trigger which in my opinion trigger for no reason is {my_host:vm.memory.size[available].last(0)}<10000M
iskorptixI'm monitoring memory usage one process which is very heavy (rabbitmq), plus that user where rabbitmq is running and all other users (including root) also are being tracked on how much of memory they use
iskorptixbut in my graphs I don't see spikes anywhere
iskorptixeven though update rate is set to be very low
iskorptix5/s
iskorptix5seconds*
iskorptixand vm.memory.size[available] is running once in 60seconds
iskorptixand sometimes (recently quite often) reporting memory usage spikes
iskorptixsometimes up to 1gb
iskorptixI tried most of the tricks, currently running out of ideas
iskorptixis there anything else I could monitor to find a proble m?
iskorptixnevermind!!!
Richlvmonologue time
Richlvalso, girls were excluded
volterWhat, where?
volterOh
mrrob941hey
mrrob941What the most amount of nodes you guys have in your zbx environment?
heajemrrob941: Until I decommissioned our Zabbix environment this month, we were monitoring close to 20k devices
mrrob941ah how come you decommissioned it Heaje?
heajeWe found that Zabbix didn't work well for us in a large environment. We were still on 2.4.
mrrob941Ya, I'm on 2.2.9 I believe.
mrrob941There are about 5 other regions, so we have about 20,000+ devices as well, but we have a lot of proxy servers, and a shit ton of storage
heajemrrob941: We have historically been a Nagios shop and Zabbix was a bit of a 4 year experiment that just didn't pan out. At this point we use Nagios for infrastructure monitoring and various teams use <name_your_favorite_tool> for metric collection and alerting
zabbixNagios is ancient like fuzzy dice, shag carpets and bell bottom jeans.
mrrob941ah
zabbixbot[04ZBX-13116] 03AWS-E-HORIZONTAL-ZabbixProxy: Zabbix busy data sender processes HIGH 14created by Manoj Kumar Popatlal14/utl568 (12https://support.zabbix.com/browse/ZBX-13116)
zabbixZBX-13116 Open Updated -1 days ago -- AWS-E-HORIZONTAL-ZabbixProxy: Zabbix busy data sender processes HIGH