Fieldysorry i blocked your message from earlier, i just saw it
{HD}Well, I successfully deleted the ACL's from a testing_dataset but, it did not seem to clear up my issue. I am now leaning towards the user to group mappings as being an issue... but, I have no idea really.
{HD}I did a test in virtual instances and I can get it setup the way I want it but I cannot seem to move those settings to my box that is 'in the field'
Fieldyit's well over my head, sorry :(
MrNazis there any way to get FreeNAS to auth users against AzureAD yet? I know you can against AD, but many clients are now ditching WinServer on prem and using AzureAD because the only thing they need on site is bulk file storage
DrKK`I have literally heard 0.000000 things about AzureAD mentioned.
DrKK`in the FreeBSD universe.
DrKK`so, that would be a very poor sign, sir.
MrNazMaybe... AzureAD provides the basic functionality of a Windows Server, but without needing an actual server. It comes free with Office 365 too, so many clients already have it.
MrNazOffice 365 + FreeNAS would be the idea setup for a large number of business use cases, because you get the benefit of central user management without the hassle of needing a windows server
MrNazAnd let's face it, Windows servers are a HUGE hassle
Fieldysounds like a plugin idea (maybe?)
MrNazyea a plugin would be good
MrNazAzureAD provides easy auth hooks so it wouldn't be hard
DrKK`you guys know plugins are gigantic pains in the ass right
Fieldyi didn't. but it's not a feature i need anyway. i stay as far away from windows stuff as possible
MrNazFieldy i'd like to avoid windows, but as an MSP, saying i'd like to avoid windows is like a mechanic saying he'd like to avoid toyota
MrNazprobably worse than that actually
peercehey, my mechanic only works on mercedes :D
peercemy other mechanic only works on toyota and nissan trucks
MrNazif you're rich enough to use the kind of mechanic that only works on mercs, then this conversation among working men is beneath you :P
Fieldybeing specialized at stuff can be a good thing. my career is network security and security systems engineering, windows is just a "LOL HELL NO" thing that's all. nothing personal
peercei have a 25 yr old mercedes. I wouldn't let anyone who isn't intimately familiar with older mercedes touch it
MrNazooo nice
DrKK`you know what I want, is a 1988 Yugo.
DrKK`that would be the tits
peerceYugo walking.
peercean eastern yurp copy of an italian econobox, what could possibly go wrong?
DrKK`there it is
peercemy car:
MrNazSo... if i were to propose to pay someone to extend the AD auth features in FreeNAS to include AzureAD... is that something that would be worth investigating or is it too fringe to be done for any reasonable cost?
peerceDrKK`; look familiar?
DrKK`should it look familiar?
DrKK`you mean it's the same as the yugo?
MrNazAzureAD is going to become the most popular directory service within a few years... MS want to migrate small business to it from on-prem servers
MrNazok so i'm trying to get a bit of interest in it as you guys can all tell
MrNazaaaand to summarize the feedback im getting:
DrKK`It's hard to get stiff wood for active directory services, sir.
DrKK`it's in the nature of the beast.
mrelceeso close.. looks like freebsd smells like it.. doesn't use the same abi, so my local poudriere is useless for the jails
MrNazyea my creed is linux and open source and so i know trying to build interest in MS integration here is like trying to build interest for sobriety at an australian football match
MrNazthanks guys... take care
DrKK`I want to meet the person
DrKK`that made this
mrelceeokay so building for freenas on freebsd is a thing.. but it seems to be all or nothing. ick.
{HD}Well, I don't think I have ever been this frustrated with an appliance before. haha. Makes me think "why do I care about this data" "I think I could continue living/working without it"
DrKK`^^^ using RAID controller with ZFS
mrelceeit could be configured for JBOD, or he's a fool. :)
DrKK`looks like the govt is shutting down
siixDrKK`: did you think they had it or were you expecting them to continue squabbling?
DrKK`not sure
DrKK`shut it all down!
cymikeACTION shutdown -h now
cymikeACTION &
DrKK`can anyone confirm, that in the Storage->Replication Tasks screen,
DrKK`that a status of "Up to date" is code for "Dude, nothing needed to be updated last time I checked",
DrKK`and that a status of "Succeeded" is code for
DrKK`"Dude, there was a snapshot that had to go, which I did, and it was successful"
DrKK`and that functionally, therefore,
DrKK`"Succeeded" is really no different than "up to date"
DrKK`except that it indicates that it actually had to update itself.
DrKK`but in both cases,
DrKK`we are up to date.
DrKK`yes or no.
siixthat is how i read it, yes
siixit should say either "There were no updates and I changed nothing and you are probably just as safe as you were before." or "There was an update and I grabbed it and installed it automatically and if something breaks now then this is probably why."
siixbut twitter designed the UI so they had to shorten it up a bit
VallHello everyone
VallI'm about to bring up a nice pool here consisting of 16x4TB Seagate Ironwolf HDDs and 2x2TB Samsung Enterprise SSDs, and there's a couple of things I'd like to bounce off you folks if you are up for it.
VallFirst thing is, I'd like to use these 2 SSDs for both ZIL and L2ARC. And using an entire SSD for ZIL and the other for L2ARC, which is what I understand the FreeNAS "create volume" WebUI component lets me do, would be a real waste.
VallThe ideal setup would be to partition those SSDs in a very small ZIL partition and a very large L2ARC partition. I gather I have to do the partitioning from the command-line, right? But and the pool creation, does it have to be from the command line too?
Cpuroasthow much RAM
Cpuroastand what's your actual workload
CpuroastSLOG won't do anything if you don't have any sync writes
VallHail Cpuroast, and thanks again for jumping in to help me.
Vall32GB RAM
Cpuroastwhat's actually going to be hitting the box, VMs?
Cpuroastor just general storage
VallWorkload will be SMB serving of a very large directory tree to a bunch (10-50, varying over time) Windows workstations
CpuroastSMB doesn't do sync writes
Cpuroastunless you enable sync=always on your datasets
Vallyeah, but I like having ZIL as future-proofing. Who knows what the people who call the shots will actually try to do in the future...
VallThey might want a VM with a database, or an iSCSI serving, or whatever
Vallthe above is just the *initial* worload
Cpuroastyou can add the SLOG at any moment in time
Cpuroastit doesn't have to be there at creation
Vallnot really, the machine is topped up in terms of bays and whatever
CpuroastSLOG and L2ARC and be added and removed whenever you want
Cpuroastat 32GB, I wouldn't bother with L2ARC
VallI mean I won't be able to add more devices.
Cpuroastadd more RAM
VallCan't add more RAM, this is the most that this machine can comport.
VallIt's an Intel S1200V3RP motherboard.
Vallyeah, unfortunately.
Cpuroastold haswell
Cpuroastwith DDR3
VallBut reportedly rock-solid and stable with FreeBSD
CpuroastI hope that wasn't purchased recently
VallIt was on sale ;-)
CpuroastSkylake/Kaby Lake
Cpuroastwould have been better
Cpuroastwith 64GB max
Cpuroastyeah, no, I wouldn't do L2ARC with 32GB
Cpuroastit only starts being interesting with 64GB+
CpuroastL2ARC isn't free
Cpuroastit uses RAM to keep track of it
Cpuroastwhich decreases RAM available to ARC
CpuroastARC(L1ARC) is obviously faster
Cpuroasthow big are the SSDs?
VallReally? Better to *not* have 4TB of flash so as to not give up a part of less than (because there's the OS, etc) 32GB of RAM?
VallThey are 2x2TB
CpuroastI'd make a 2x2TB mirrored pool with that
Cpuroastand use it as fast storage
VallThere's 4x2TB SSDs in another pool for the "fast" part
Cpuroast4TB of flash would eat quite a bit of the RAM
Cpuroastin L2ARC
Vall(I didn't mention them or this other pool because they are not really my main concern right now)
Cpuroastwhich would actually decrease performance
VallOK, so you actually think it's better to forgo the 4TB of L2ARC SSD cache in order to save as much as possible of the 32GB of RAM for L1ARC?
VallThis is *very* anti-intuitive for me
VallI mean, SSD is slower than RAM, no doubt... but still beats the hell out of HDDs
CpuroastL2ARC gets warm blocks ejected from ARC(L1ARC_
VallYeah, I know. So the block has to come though the ARC RAM first.
CpuroastVall: sure SSDs are fast, but if you ear a large chunk of your RAM, your ARC will suffer
VallBut all and every block read from the HDDs will, no?
Cpuroastall blocks have to come from HDDs anyways
Cpuroasthot blocks will stay in ARC
Valland when the ARC is full, which I understand will be all the time, they will be moved to L2ARC
VallWhich there will be a *lot* of
Cpuroastbut then it will start decrease RAM
Cpuroastavailable to ARC
Cpuroastmaking things work
Cpuroastyou can try
CpuroastI think 2TB is too much ARC
VallSay, in the most extreme case, that ARC is reduced to just 1 block.
CpuroastI mean
VallThis means every block read from HDD is evicted to L2ARC immediately
Cpuroastif you absolutely want L2ARC, something like 256GB, maybe
Vall(immediately when the next block is read)
Vallbut then it stays there indefinitely, being served from there instead of HDDs
slug_frHi everyone
CpuroastI think performance will suffer
Cpuroastwith too much L2ARC
VallI can do a benchmark once the data is migrated.
CpuroastI would never recommend L2ARC on anything less then 64GB of RAM
VallAnd remove the L2ARC later if it doesn't work
slug_frI got an LSI-SAS-9210-8i-8-port-6Gb-s-PCIe-HBA-RAID-SATA-Controller-card-M1015-9211-8I on ebay and a very similar controller (same chip) runs in my other HP Microserver gen8 without any issues... this one leads to a "nmi detected please consult the integrated management log for more details”
Cpuroastand even then not several TBs of it
slug_frwith the card inside, I don't even get into the bios anymore
VallWeird, CPURoast. Because ZFS (with ARC and L2ARC) exists since *much* earlier than machines with 64GB RAM...
slug_frI disabled the internal controller and also set the "Embedded SATA Configuration" from "Enable Dynamic HP Smart Array B120i RAID Support" to "Enable SATA AHCI Support"
slug_frany ideas?
VallAnd I've been using L2ARC with very measurable success on a machine with 16GB RAM (I have the benchmarks before and after adding 256GB of L2ARC, and the numbers skyrocketed 5 to 8 times as much)
Cpuroastyeah, 128-256GB sure
Cpuroastbut not 4TBs of it
Cpuroastits all about balance
VallCpuroast: I understand.
VallPerhaps, to follow your recommendation, I could partition those SSDs in 3: one very small area for ZIL (I still have to do the math, but the rule is still just enough for 30secs of sustained writes to the main HDD pool, right?), a slightly larger area for L2ARC (say, 256 or 512GB) and the rest to be *added* to the other fast pool.
VallCpuroast: what do you think?
CpuroastI wouldn't bother with those 2TBs for L2ARC of SLOG
Cpuroastadd em all to the fast pool
Cpuroastand expand it
Cpuroastmaybe get a smaller NVMe PCIe SSD
Cpuroastfor cache purposes
VallThe fast pool is already as large as needed for the data it will contain
VallAnd I have no free PCIe slots unfortunately
Cpuroastoh fun
Vall(there were three, and they are being used for 2 SAS and 1 10GBps Ethernet controllers)
Cpuroastyeah, for your usage, I would have gone with a Xeon E5
Cpuroastwhich provides more PCIe lanes
Cpuroastmore RAM
Cpuroastand much more scalability
Cpuroastanyways, the box should do the trick for the most part
CpuroastVall: partition off 1 of the 2TB to 128-256GB
Cpuroastand use that for L2ARC
Cpuroastand forget the rest exists
VallCpuroast: I understand. My idea is ti play the best game I can with the cards I've been dealt
Vall*ti -> to
Cpuroastthen partition off the other 2TB to also 128-256GB
Cpuroastand and use that for SLOG
Cpuroastand forget the rest exists
Cpuroast128-256GB shouldn't be too damaging to RAM
VallCpuroast: partition is best (or only) done via gpart on the command-line, right?
Cpuroastand leaving the rest of the SLOG unpartitioned should improve write latency consistency
Cpuroastas the SSD will have a VERY large amount of empty NAND to absorb writes
Cpuroastyeah, partitioning is only handled at the command line
VallGreat. So here is my idea:
Vall1) Create the pool with the GUI and *only* the HDDs
Vall(by the way, I'm structuring it as 2 raidz2 slices, each with 8x4TB drives -- 16 drives total)
Cpuroast2 vdevs
cbreaka single raidz3 would be a viable alternative. But less performant, different reliability characteristic and slightly bigger
Vall2) Via the command line, partitition the 2 SSDs as you recommended (is gpart the best tool for that?)
cbreakand less flexible
Cpuroastcbreak: not with 16 drives
Cpuroastno way
Cpuroastway too wide
cbreakCpuroast: I've heard that people managed much much more :)
Cpuroastdepends on workload
cbreakas always :)
CpuroastIOPS will be worse
Cpuroastand rebuild times will be insane
Vallcbreak: thanks for the input, but I'd rather have 2 vdevs to parallel-distribute reads and writes over.
Cpuroast2 vdevs = double the write IOPS
Cpuroastand shorter resilvers
VallAnd then there's the resilver times as CPUroast pointed out
cbreakhave you tried the new resilver code?
VallCPUroast: is gpart the best command-line tool to create those partitions on the SSDs?
cbreaknot sure if it arrived at bsd yet
Cpuroastcbreak: oh sure, the mostly sequential resilver will obviously help
VallCpuroast: so it is obviously not that: "The gmap program computes, in a sequential way, a static mapping of a source graph onto a target graph. The gpart program is a simplified interface to gmap (...)"
cbreakor gpt
cbreakif you have a mac you can use it via the gpt tool
cbreaklinux will also do
cbreakmany options :)
Cpuroastwhy bother
Cpuroastfreebsd has everything needed
CQhello, I have a freenas box sitting in a rack, and am wondering if it makes sense to put a jail or VM onto it with Xwindows to be used remotely... up until now, all th esolutions I've tried (X forwarding, VNC; etc.) have seemed a little clunky and unresponsive. I'm running gigabit ethernet
CQright now these are VMs on my desktop, but I'd rather have them on a headless server with a remote display
cbreakwhy remote display?
cbreakxforwarding should work via ssh... as long as you only use X
CQbecause the NAS box is headless and I need to see something ... ?
cbreakCQ: why?
cbreakwhy do you need to see something?
cbreakwhat do you want to see?
CQI have a VM that I only do internet banking on for example
cbreakdoesn't sound like something that could be improved by putting it on an always-internet-connected device
CQa better example is that my MP3 collection is on that machine, and doing mass tag changes is very slow if remote
cbreakif you want to install software on freenas to run on freenas, then a jail is the right choice
CQso your suggestionis SSH with X forwarding...
cbreakbut GUI software... maybe not a good idea
cbreakx forwarding is the simplest way I think
CQthat's why I was thinking VM, that way the networking is only local
cbreakif you even need a GUI
CQI'll play around with it
cbreakall jail software I used so far was command line based, or had a web based UI
Vallcbreak: linux is actually (still) my main bread & butter here, but FreeNAS as a appliance, with an almost-enduser WebUI interface available, leads me to forgo that course ;-)
cbreakwell, freebsd is a proper unix
VallSo, is gpt or gparted better to partition on the command-line? I'm mostly used to Linux' fdisk...
cbreakstill, modern linux distros are probably quite similar
cbreakgpt is annoying to use
Vallso gparted is less anoying, right?
cbreakI think so
Cpuroastwhy not simply use gpart
Cpuroastas in built-in to freebsd/freenas?
VallIsn't it a program that "computes, in a sequential way, a static mapping of a source graph onto a target graph"
Cpuroastas in, why bother with external tools?
Cpuroastit's the partitioning tool freebsd uses
VallWhere can I find docs on it?
CpuroastI already linked you 2
VallBecause it's not the URL you linked above
Vallthe "(8)" wasn't making it into the link, sorry about that
Cpuroastessentially it would be something like "gpart create -s GPT ada1"
Cpuroast"gpart add -s 128g -t freebsd-zfs ada1"
Cpuroastreplace ada1 with whatever device your SSDs are
VallThanks Cpuroast, your help is great as usual, and much appreciated.
Cpuroastthen you would add the device from GUI
Cpuroastto the pool
Cpuroastlog= SLOG
VallDoes the GUI lists partitions as possible options to add to the pool?
Cpuroastit might
VallI was under the impression it didn't
CpuroastI'm not sure
VallI will try.
VallIf it doesn't work, I will export the pool in the GUI, import and add the devices and export on the command-line, then reimport it on the GUI
VallOr is there a better course?
Cpuroastyeah, that works
VallThanks again
VallGoing down onto it right away (well, after a cup of coffee...)
Cpuroastyou would add the devices, export the pool
Cpuroastand then import
cbreakwhy the export / import?
cbreakzfs can deal with added log / cache devices at runtime
Cpuroastfreenas UI can't
Cpuroastif FreeNAS UI doesn't allow partitions to be added
Cpuroasthe'll to add via CLI
Cpuroastand since CLI won't be reflected in GUI
Cpuroastunless export/import
cbreaksooner or later he'll install an update, which requires a reboot...
Cpuroastor that
CpuroastI kinda wish FreeNAS would allow size reduction for L2ARC devices
Cpuroastin UI
Cpuroastlike pick device, give it size, done
CpuroastSSDs are getting bigger and bigger
cbreakyou're worried about the l2 lookup table size in ram?
Cpuroastand you really don't want massive amounts of L2ARC for systems with 16-32GB
Cpuroast2TB of L2ARC fully warmed up would take quite a bit of RAM
cbreakmaybe there's a tunable for that
Cpuroastdoesn't look like it
VallCpuroast, cbreak: back from coffee (or should I say, covfefe? ;-) ) and had a look at your very interesting discussion above.
VallIn ZoL there are ZFS module parameters to control what goes into ARC and L2ARC.
VallI think it can be specified to both, to only hold metadata. Would that help?
Vall(in the case of [L1]ARC, of course)
Cpuroastdefault is fine
Vallhaving a look at the ZoL parameter list
Vallyeah I don't see anything that could help.
Vallgetting back onto the grind (create main pool, partition SSDs, add the partitions, etc)
VallThanks CpuRoast and cbreak.
Vallcbreak: out of curiosity, does your nick come from the terminal mode? ;-)
VallCpuroast: out of curiosity, what does the "pass NN" at the end of each line of "camcontrol devlist" means? Simply the order the drives were probed during boot?
cbreakVall: no, it has a much lamer origin
VallOK... I had thought it really sleek ;-)
cbreakyes. I kind of retconned it in
Helphello !
Guest53506Someone can help me... ? :)
VallCpuroast, waiting for my pool to get populated for some testing... meanwhile, been mulling the L2ARC sizing vs. available RAM issue.
VallCan you confirm whether this formula is correct:
Vall (L2ARC size in kilobytes) / (typical recordsize -- or volblocksize -- in kilobytes) * 70 bytes = ARC header size in RAM.
CpuroastVall: sounds right
Cpuroastso if you were to give an entire 2TB SSD as L2ARC
VallThanks Cpuroast.
Guest53506is it possible to configure a DHCP on FREENAS ?
VallI did the math here...I would need:
cbreakGuest53506: freenas uses dhcp if you want, sure.
cbreakI'd configure it statically though
Guest53506I can't find the option on my freenas :(
Cpuroastyou'd end up with ~31.83GB
Cpuroastif it was fillex
Cpuroastwhich would eat your RAM alive
Cpuroastwhere as 128GB of L2ARC would be around ~2.03GB
Cpuroastwhich is much more reasonable
Cpuroastyour ARC drops by 2GB, you gain 128GB of L2ARC
Cpuroastthat's manageable
Cpuroast256GB would mean ~4.07GB
Cpuroastwith 32GB of RAM, I wouldn't go beyond 256GB
VallCpuroast: my calculations show an even grimmer picture than yours
Vallfor 2TB of L2ARC, I find
Cpuroast2,000,000,000,000 / 4096 * 70 bytes)/1024^3=~31.83GB
VallWhy 4096?
VallIsn't ZFS recsize tipically 128KB?
Vall4KB is actually the absolutely worst case, no?
Vall(as I'm working with ashift=12)
CpuroastZFS automatically adapts the record size dynamically to adapt to data
Vallincluding compression effects etc
Cpuroastso 4096 is worst case scenario
VallI have a populated dirtree here. Can I somehow find out what the average ZFS blocksize for it is?
Vallpool01/usr2 584G 90.4G 524G /usr2
VallIf I can obtain average data from it, I can refine the calculation
Vallusing the absolutely worst case for *average* record size doesn't sound like a good strategy
FRIGNHey guys
FRIGNdo you think 43°C max is too high for HDD-temps?
Cpuroastit's ok
Cpuroastprobably at the upper limits
Cpuroastis about right
FRIGNI am normaly coasting around 37°C but when I have a smart-test or scrub it goes to 43
FRIGNCpuroast: ok thanks man :)
CpuroastVall: so the min is most likely 4K and the max is 128KB
Cpuroaston FreeNAS created pools
Cpuroastit's quite easy to guestimate, look at your average file sizes
Cpuroasttons of small files, you'll hit 4K more often then not
Cpuroasttons of large files, you'll hit 128KB most of the time
cbreakdoesn't zdb have some histogram function?
cbreakzdb -b
Vallcbreak: good catch
Vallzdb -b pool01/usr2
VallDataset pool01/usr2 [ZPL], ID 38, cr_txg 110, 524G, 1194397 objects
Vallecho 524 1024 \* 1024 \* 1924 \* 1194397 / p | dc
VallSo each "object" (file, dir, etc) is an average 885KB (decimal KB) in size
VallSo average recsize on that ZFS dataset would tend towards the dataset recsize, which is 128KB, correct?
Vall(as a consequence of 885KB being *way* larger than 128KB)
VallOr should I look for a more nuanced estimate, considering ranges of sizes for files etc?
SwKwhats the recommended m.2 nvme for slog these days?
ZewwyI know you can add vDevs to a pool, but there's no way to tell the Pool to relocate all data from a vDev to all other vDevs and remove the vDev that was initially added, eh
Zewwyyou'd literaly have to migrate all data, and rebuild
CpuroastSwK: enterprise or home use
VallCpuroasẗ: any further thoughts on what I posted above (starting with "zdb -b pool01/usr2")?
Cpuroastall depends on how ZFS allocates records
Cpuroastnot sure how it handles end of files
Cpuroastso say, 885/128=6.9140625 blocks
Cpuroastso I assume it would store 6x128
Cpuroastthen reduce for the last block
Cpuroastcompression has to be factored in also
Cpuroastwhich would reduce the record size
Cpuroast(2,000,000,000,000 / 131072 * 70)/1024^3 = ~0.99GB
CpuroastI really doubt it will be the best scenario
Cpuroastbest case scenario
Cpuroastprobably somewhere in between
nanopetehello there, Im on a stable freenas train, I just deleted an iocage jail through the new UI, but that seemed to take a couple of other iocage jails with it that I did not want to delete
nanopeteno worries I should have periodic snapshots of /mnt/iocage and recursive
nanopetebut I cant find the iocage jails in there either?
nanopetedoes anyone have any suggestions_
SweetAndLownanopete: most people haven't used iocage much yet
SweetAndLowi doubt it would delete all your jails but it could if there was a bug
FahadHi everybody, I've been running FreeNas on a nice purpose built fileserver for a couple of years but it's time for some housekeeping and I need some support
FahadFirst off I'm still on 9.10, and I'll get to updating after I sort out my disk semi-issues, though kinda anxious about the process
cbreakFahad: it was fairly easy for me
cbreakI went from 9.10 to 11
cbreakI don't remember major problems
FahadI'm running a big pool of 6tbx10 on zfs2, and two of the disks have been showing "Device: /dev/da0 [SAT], XX Currently unreadable (pending) sectors"
cbreakbut I think some jails required some attention
cbreakconsider replacing them
cbreakdo you have additional hard disk bays and sata ports?
FahadI've been ignoring it for a while and the number is small and hasnt been increasing, but I'm willing to replace them to be safe
Fahad15 bays, all hooked up, but the second pool of 5 can be taken offline maybe until I resilver?
SweetAndLowFahad: do not ignore failing disks? why would you do that
SweetAndLowreplace them
FahadThey have been on a very low number since the error showed up and some people have shrugged it off. But I'll replace them now that I've gotten spares.
FahadWhat is the recommended process for replacing the disks? and any precautions I can take for the case of large disks like the 6tbs?
cbreakFahad: add a new disk to the computer
cbreaktell freenas via the web ui to replace one of the failing disks with the new one
cbreakrepeat with the second disk
cbreakwait some more
cbreakwhen you're done waiting and the nas is done resilvering, you can remove the old hard disks
FahadShould I do both simultaneously or in sequence?
cbreakdoesn't matter much as long as the old hard disks stay in the pool I think
cbreakit might be bad for resilver performance to do two at once though :)
cbreak(unless zfs is clever about it)
FahadThen I'll just do them 1 by 1
sunrunner20Fahad, one at a time
sunrunner20esp if you're on 11.1
FahadHow long of a wait should I expect for 6tb resilver? order of magnitude.. 24 hours?
cbreakyou are very optimistic
sunrunner20drive size is irrelevent
FahadI'm on 9.10
sunrunner20its how much data you have
Fahad45TB :eyes:
cbreakmaybe a week... or three?
FahadOK so I take a deep breath
SweetAndLowFahad: read the manual
Fahadand consider this a construction project
cbreaktook me about two days to resilver 9tb or so
Fahadencrypted or not? mine aren't
cbreakdoesn't matter
cbreakencryption is below zfs in freenas
cbreakmine were encrypted, but I have an atom cpu
cbreakso more than capable to encrypt that data
cbreak(and decrypt it)
cbreakthis isn't really CPU bound, it's IO bound
FahadFor my other pool of 5 hdds, can I just remove them physically while it's off?
FahadThey're currently empty. Set up for a pool, but no data resides on them \yet.
cbreakconsider exporting the pool
cbreakbut mind the checkbox
cbreakI think there's one that causes freenas to destroy the pool
cbreakbut I don't remember if having it checked or unchecked destroys it
FahadWhat do I get out of exporting it?
FahadI just need to empty the bays to use them for resilvering
cbreakFahad: freenas won't miss the pool
cbreakyou won't lose any data
FahadWould I lose anything if I just remove them, and put them back in sometime later?
sunrunner20resilvers should be pretty close to the sequential write speed of the drive
sunrunner20Fahad, you'd lose all redundancy
sunrunner20zfs wouldn't have to do a complete resilver but it would have to bring them current again
sunrunner20and you have to zero ZFS drives before reusing them in another pool
FahadI see. Well, the other pool I'm taking down will just be set aside, and I'll be resilvering the problematic disks with brand new ones
sunrunner20if its just going offline then just export it
SweetAndLowFahad: did you burn in these new drives?
SweetAndLowthere is a good chance they will fail during the resilver if they are bad
sunrunner20and you get a safe offline out of exporting the pool
FahadIt was more than a year ago so I can barely remember.. But they are practically empty
sunrunner20the need to replace bad drives with new ones faaaar outweighs any advantage of burning them in imho
FahadI didn't get a chance to use them after installing
SweetAndLowif they fail during replacment then why bother
sunrunner20your odds of a bad drive are <1 in 50
FahadThey're HGSTs so I have high confidence too
FahadSo far so good with all of them
sunrunner20SweetAndLow, because if they don't fail you just got your redundancy back a week sooner
sunrunner20the choice is use them and esentially burn them in with the resilver process or burn them in then resilver
sunrunner20the chance of failure of the new drives is equal in both cases
sunrunner20but in one case you have your pool back shipshape a week sooner
sunrunner20and Fahad if you haven't done it yet. Follow the manual to backup your encryption keys and TEST the restore procedure
sunrunner20and be sure you're on the latest update
sunrunner20it's been a couple months but I think 9.10 had an issue where it'd destory the pool/geli info importing the pool
sunrunner20pissed me off that iX took 15 days to push a fix
SweetAndLow9.10 is fine
sunrunner20AllanJude submitted a fix within hours
SweetAndLowthat but was in 11.0
SweetAndLowthen 11.1 had the scrub hang and memory leak issue
SweetAndLowthen 11.1-U1 seems to be ok for the most part
sunrunner20pretty sure it made its way into a couple 9.10 images
sunrunner20I've had no issues with 11.1
SweetAndLowsunrunner20: you on u1?
SweetAndLowor just .1
sunrunner20don't think so
SweetAndLowyou should definatly upgrade to u1
sunrunner20let me check
Fahad@sunrunner20 No encryption keys if I'm not encrypting drives, no?
sunrunner20but I thought you said the pool was encrypted
sunrunner20SweetAndLow, yea flat 11.1
sunrunner20I'll update tonight though
sunrunner20need to update Plex too
Fahad After I replace the problem drives I'll probably update and have to do jails from scratch
sunrunner20my jails came over fine from 9.10 to 11.1
sunrunner20haven't tried creating a new one though
SweetAndLowsunrunner20: jails in 11.1 are kind of broken
sunrunner20that probably IS broken
SweetAndLowthe are stuck on 11.0 and ports will not biuld correctly because 11.0 is eol
SweetAndLowthe warden jail template never got updated
sunrunner20what is iX doing?!
SweetAndLowi think there is way more focus on iocage
sunrunner20honestly I think it may be time for me to flip to plain freebsd or ZoL
mgolischno idea
cbreakI built a new nas from scratch for work
cbreakmy freenas works fine for home use
cbreakbut at work I use ubuntu...
cbreakseems more reliable
cbreakand administration is nicer
SweetAndLowsunrunner20: or contribute fixes and code
cbreak(I use something called ansible)
sunrunner20I've tried several
SweetAndLowlinux just doens't have what it takes for a nas appliance at this time
sunrunner20every PR i've submitted has been rejected
cbreakwell, at work I don't need an apliance
cbreakat home I use one :)
sunrunner20I think ansible is what AllanJude uses
cbreakthat name seems familliar
cbreaksome ZFS dev?
sunrunner20it should
sunrunner20world class ZFS SME
cbreakI barely remember the names of my coworkers
sunrunner20and freebsd dev
sunrunner20lots of zfs focus
mgolischdid 9.10 have iocage?
mgolischi dont remember when i changed all my jails to it but it worked rather well since then for me
AllanJudesadly puppet mostly
AllanJudenot ansible
sunrunner20my mistake AllanJude
sunrunner20i've been learning Chef
sunrunner20if I have to redo my VPS i'm doing it with Chef
sunrunner20can't login to my WP admin
sunrunner20half suspect its been hacked
sunrunner20but CPU and bandwidth consumption are nonexistant
SweetAndLowmgolisch: no, iocage came with 11.0
sunrunner20how's zfsd coming along AllanJude ?
AllanJudesunrunner20: I don't really work on zfsd
AllanJudei have plans for the way FreeBSD lays out ZFS disks
sunrunner20that must be another false memory
AllanJudeto make it work better with zfsd
sunrunner20I have too many of those lately :(
AllanJudeto make the hot-spare feature work even if you boot from the zfs disk etc
sunrunner20zfsd handles replication too doesn't it?
sunrunner20whats the offical page for zfsd? google only turns up for me
VallCpuroast: Thanks for the final comments re: record size for calculating L2ARC memory usage
VallAnother question: this machine has 2 10Gbps adapters. What is the best way to aggregate them, when they are connected to Cisco switches?
Cpuroastfor SMB?
VallLACP does not do any balancing when both interfaces are up, right?
sunrunner20but single clients won't see an improvement
sunrunner20unless they changed that in SMB3
Vallsunrunner20: exactly
CpuroastVall: it does do balancing
sunrunner20or is it 3.1?
CpuroastVall: with multiple clients
VallCPURoast: OK
Cpuroastso if you have 50 clients hitting the server
VallThe problem is when it talks with another server
Vall(which is basically a big SMB client)
Cpuroastyou should get a relatively even split
VallCPURoast: there's an option on FreeNAS "aggregation" screen called "Load Balance". So it doesn't work?
Cpuroastthe only way to load balance single clients
Cpuroastis MPIO
CpuroastI think SMB3.1 has MPIO
Cpuroastand MPIO is in NFS v4.1
VallFreeNAS in itself supports SMB3.1/MPIO?
CpuroastSMB Multi-channel
Cpuroastnot sure
Cpuroastif FreeNAS's Samba implementation handles it yet
sunrunner20you'd have to compare when samba implemented that and see if it lines up with FN's version number
mgolischalso that would probably only benefit you if that server(client) was a windows machine
mgolischiam quite sure that the smb/cifs filesystem drivers in most other oses dont support this feature
Vallfolks, I've configured an LACP LAGG in FreeNAS Interfaces/Aggregation menu, aggregating interfaces ix0 and ix1 (my two 10GBps interfaces). But when I pressed OK, the machine lost all network access.
m0nkey_Yeah, the documentation does say it'll do that.
Vall(I was of course accessing the machine over *another* interface, ixgb0)
VallChecking the interfaces with "ifconfig -a", ALL of them (not only ix0 and ix1) have the IFDISABLED bit set.
VallRebooting the machine (all I could ask the local person to do, as I'm remote)
Cpuroastyeah, lagg usually requires a reboot
VallRebooting the machine fixed nothing: ALL interfaces are still with IFDISABLED.
VallSo I have no access to the machine.
Cpuroastlagg creates a new iterface
Cpuroastyou assign IP to this new lagg0 interface
m0nkey_"Creating or editing link aggregations can disconnect clients using the FreeNAS® computer. Please verify that clients have saved their work and are not connected through the affected networks before making changes."
Vallso it should be further on in "ifconfig -a" output, let me advance
m0nkey_Vall: If you have an IPMI, jump onto that and finish configuring the LAGG through the console.
Cpuroasta vlan creates a virtual interface a lagg creates a virtual interface
Vallno IPMI, unfortunatelly
VallAll I have is a RMON
Vall(Remote Moronic Operating Nerd)
Cpuroastcreating a lagg shouldn't have affected igb0
VallWill check ifconfig output and look for these interfaces
Vallyep, my thoughs exactly
CpuroastI know it didn't when I configured my 2x 10Gbit Lagg
VallIn my case, it's Murphy's Law
Cpuroastdid you configure lagg on the switch
Cpuroastfor those 2 ports
Vallthe RMON is crazy to leave (friday before a BIG holiday here)
VallNo, I did not configure LACP on these two ports. Was going to do it *after* configuring it on the NAS
Cpuroastconfiguring an LACP lagg requires server config and switch config
Cpuroastusually best to do it before
Cpuroastso the lagg comes up
VallBut should not have affected igb0 anyway, right?
Cpuroastigb0 should be still up
Cpuroastwithout any issues
Vallyeah, but it's showing IFDISABLED among the flags
Valler, nd6 options
Vallyeah, my thoughts exactly
Valllagg0 is in fact created
Vallbut shows IFDISABLED in nd6 options too
Vallperhaps FreeNAS added *all* interfaces to lagg0? how can I check?
CpuroastIFDISABLED is actually normal
FahadWhat's typically fastest way to copy files from FreeNAS to a windows device? For one time not for practicality
CpuroastI checked all my interfaces
Cpuroastand they all have it
FahadFTP? Assume that'd be faster than SMB
CpuroastFahad: no
CpuroastFTP is actually slow
CpuroastSMB is probably your best bet
VallI haven't noticed IFDISABLED before
Cpuroastit's there
Fahadfair enough
VallCpuroast: any way to check whether LAGG0 by error is including all interfaces? that would explain everything
Cpuroastyou can reset the network from console
Cpuroasthence IPMI is useful
VallSo, from the end of ifconfig lagg0 output, I see ONLY ix0 and ix1 listed among laggport
Vallso indeed igb0 should not have been touched
Vallbut it lost all conectivity
Vallit doesn't even show as UP in its first "ifconfig" line
Cpuroastshould look like this
Cpuroastminus the VLAN stuff
Vallwhat is the command to reset the Network?
Cpuroastit's one of the options
Cpuroastin the console menu
VallOn the same menu my RMON entered 12 to go to the Shell, correct?
Cpuroastyou have a network item
Cpuroastin the menu
VallOK, telling my RMON to activate it
DarkainMXon my 10gbe interfaces, I can pull upwards of 5gbps+ from FreeNAS to Win7. The limiting factor was disk write speed on my WIn7 box. No fancy configs at all. Just stock SMB.
DarkainMXServer was an old dual-proc Xeon 5400 series. Client is a newer Xeon E5 workstation.
mgolischyeah but that requires 10gbe connections on both ends
DarkainMXYup, which are dirt ass cheap now.
DarkainMXI eBayed the cards for what... like $15-20/ea? Then another $5-10 ea for the SFP adapters for each. Forget what I paid for the fiber, but from desktop to switch was a longer run so not the cheapest.
mgolischyeah but 10gbe switches are not that cheap
DarkainMXI paid what... $50-ish for a 48port gbe + 2 port 10gbe? and another $150-ish for a 20 port 10gbe.
DarkainMXyou can also just direct wire between the two machines if its a one-off file copy
mgolischthat must have been quite a deal
mgolischi paid way more for my lb6m
DarkainMXor just run 2 NICs (which is actually what I'm doing, 10gbe for heavy stuff, 1gbe for dev/testing)
DarkainMXI managed to snag my 5650TD for ~$50, but they're still in the $100 range. Great switch for getting started with some 10gbe mixed with existing 1gbe -
VallI'm powering everything down (FreeNAS and switches and then powering them back up again)
sysadminman if the NFS support in FN 9.10 sucks
Vallsame thing, no interfaces are uo
sysadminanyone can suggest me which versio to upgrade to?
VallMy operating hypotesis is that the switch went south
VallWhen I configured LAGG on the NAS
Vallremoving the LAGG configuration and power-cycling it all again
VallOK, just removing the LAGG was enough for all interfaces (including the ones that were NOT on the LAGG) to come back up
VallDidn't even need a power-cycle
mgolischprobably some sort of misconfiguration on the switch
mgolischstill wonder why your non lagg stuff would go down
Vallmgolish: agree on the "odd" part... ;-)
Vallhaving to run "dhclient igb0" by hand
VallWill try again reconfiguring the switch by hand
Vall( reconfiguring the switch by hand *first*)
VallHow do I add another IPv4 to the same ethernet address (ie, how to do the FreeBSD equivalent of "ifconfig eth0:1" in linux)?
peercenetwork -> interfaces -> select the network interface, edit, and add aliases to the form
mgolischon the cmdline?
mgolischno idea
peercebtw, ifconfig is deprecated in linux now, you should use 'ip addr ...'
slug_fror install that package that gives you the old commands back ;-)
slug_frforgot the name but I honestly think it's worth it...
mgolischits usualy always installed
mgolischas alot of stuff still uses it
Pixelzit can potentially show wrong information
Pixelzparticularly ifconfig and netstat
slug_frPixelz: compared to ip addr...?
mgolischss ftw
peercethe ip command is native, the old commands are emulations
slug_frphew... ok, that's a fact but it sounds more like a theoretical thing than an actual issue - any experience/excamples?
ZewwyIs it common for the updater on FreeNAS to hang out at 20%?
Vallpeerce: the day linux stops supporting plain-old-goode-and-nice ifconfig, is the day I will stop using it definitely ;-)
Zewwy(2/2) Installing packages (60%)... Installing base-os (1/5) 20%
mgolischwhy ip is so much better
slug_frmgolisch: curious!
ZewwyOoo it moved
Zewwytime to take a walk
mgolischatleast in my mind alot of the ip commands are more logical
Pixelzroute is a whole lot better for sure
mgolischand it has all that cool network namespaces stuff
mgolischbut yeah for common stuff you will probably never touch that
PixelzI don't remember examples from the top of my head but I know it can show incorrect information
slug_frjust wondering why such standard things were touched anyways....
Pixelzand it's been deprecated for a really long time
PixelzI think possibly a decade or so at this point
Pixelzit's time to bite the bullet and learn the new tools
slug_fr... but it's a fact, so you're right: the discussion is actually not worth the time discussing it but getting used to it
Pixelzit's just a matter of getting used to it really
Zewwychange is hard for some
ZewwyI'm all for change as long as its intuitive change
ZewwyMy favorite quote I've read online "A user interface is like a joke, if you have to explain it, it's not very good."
slug_frZewwy: I agree and want to add: that it's probably not ifconfig and route that are good examples for UIs explaining themselves ... it's just that they're around for so long and you didn't have an alternative that you bit into it and just learned it and now it feels intuitive but... I'd say it's probably not more or less intuitive than ip...
PLChi' there
Zewwywooo I guess the update failed
Zewwylogged back into FreeNAS and still saw I was on 9.10 base, I attempted the latest 9.10 build, but since it failed, I'm gonna try updatign to the 11 Stable build
sunrunner20afternoon DrKK`
sunrunner20gotta update my Plex tonight
sunrunner20big update to DVR features
DrKK`I might get the plexpass thing
sunrunner20to watch stuff on your phone?
m0nkey_Had Plex Pass. Decided not to renew this year
SweetAndLowwhy do people pay the monthly fee? just buy a lifetime pass
SweetAndLowI think i bought it a cuople hears back for $70
DrKK`let me ask a dumb question.
DrKK`why are there people with, say, 5 or 6 drives,
DrKK`their motherboard has 6 or 8 SATA ports on it,
DrKK`and they're INSISTING on using a RAID or HBA card
DrKK`can someone answer me this please?
SweetAndLowbecause it's better
DrKK`because I feel like I'm not getting something
m0nkey_Because reasons.
m0nkey_I took my HBA out. Wasn't being used. Drives were already on the SATA ports
SweetAndLowI think there was some reddit post that said a hba was better than your motherboard sata ports and people have been doing that for the past 6 months
DrKK`but that doesnt even make sense.
DrKK`SweetAndLow: wut
outrageousBecause those additional RAID controllers used to be considered the bees knees?
sunrunner20I don't get it either DrKK`
SweetAndLowDrKK`: yeah people kept coming in to the forums saying hba was better
DrKK`This is why people like me always will have a job and get paid good money, I guess.
sunrunner20the subreddit for freenas and digital hoarding are awful
sunrunner20full of completely incompetent bafoons
m0nkey_Back later. About to lose signal
DrKK`the subreddit for freenas is almost exclusively populated,
sunrunner20I think I'm still banned for calling a mod an idiot
DrKK`by people either Cyberjock, me, or one of the forum dudes,
DrKK`has more or less embarrassed on the actual freenas forum
DrKK`for not kjnowing what the hell they're talking about
DrKK`so a good rule of thumb is:
DrKK`do not listen to anything on the FreeNAS subreddit
sunrunner20I barely know and I run circles around most of them
DrKK`since by definition, it is almost exclusively populated by people we have rejected on the forums due to extreme lack of understanding or genetic stupidity.
ZewwyThose same people probaly think FreeNAS run son Linux
mgolischit might be if you only have 4 and need more drives
DrKK`"I have a friend who is real big into Linux IT and he says .................."
ZewwyLinux is like religion, which one do you follow?
sunrunner20I follow whichever one will get the damn job done the best
ZewwyThere are more forks flying around then in Hells Kitchen
mgolischthere is not much religion theres only one linux
ZewwyReally one linux that has multiple package managers?
sunrunner20which is usually Ubuntu or Ubuntu server
ZewwyYum, Apt-get, etc
DrKK`if you have to use Linux,
DrKK`it has to be Mint.
DrKK`everyone knows that.
ZewwyI'm a Ubunutu guy
ZewwyUsed to be a Debian guy
sunrunner20although didn't yum replace rpm?
DrKK`stop discussing linux in the #freenas channel
SweetAndLowjust pick the number 1 entry on distrowatch
mgolischyum is a frontend to rpm
DrKK`lest you open a black hole singularity
mgolischlike apt is to dpkg
ZewwyWhats BSDs package manager again?
SweetAndLowdnf replaced yum
ZewwyLets see how this 9.10 base to 11.1 upgrade goes
ZewwyI have a feeling it'll fail
PLCI've got new log from /var/log/message : Feb 9 21:15:46 freenas [common.pipesubr:66] Popen()ing: /usr/local/sbin/dmidecode -s system-product-name and Feb 9 21:15:46 freenas [common.pipesubr:66] Popen()ing: /usr/local/sbin/dmidecode -s baseboard-product-name
PLCwtf ? :D
SweetAndLowPLC: system is on fire
ZewwyON FIRE!!!!!
sunrunner20I forget the story behind that error
sunrunner20PLC, its a error in linux iirc
SweetAndLowthis log message was introduced when there was that watchdog fix for the asrock boards
PLCohhh it's a fix message log ^^
SweetAndLowi currently have 100+ lines of that in my logs
SweetAndLowpretty much the only log messages i get
SweetAndLowshould happen once a hour
DrKK`that's a normal log message
DrKK`total log spam
DrKK`it's the FreeNAS monitoring itself
DrKK`I really wish they would get rid of that shit
DrKK`back later, have to take child to library
sunrunner20at least its not that log spam you get in 11.1 every minute
PLCokok no prob :)
PLCI' m looking for a brick/fail acces (brutforce or anything like that) on my freenas
sunrunner20its a single line fix that somehow wasn't considered important enough to fix before the 11.1 release
PLCouaw :p
sunrunner20mmm no such thing unfortunately
zaggynlquestion: why is the recommended capacity for a zfs volume <= 80%?
sunrunner20zaggynl, ZFS changes block allocation algorithms somewhere between 80 and 90%
SweetAndLowanything is fine but at 90% performance starts to decrease
PLCI had got a very big fail at work this afternoon
sunrunner20the different algo is much slower
zaggynlI guess I should make this read only then
sunrunner20and fragmentation increases
PLCand SI manager came see my workstation … :p
sunrunner20also a 100% full ZFS volume is difficult to recover
PLCa virus was putting the brothel from my post
PLCnot post … workstation ^^
sunrunner20we highly recomend a dataset with a reservation of a few gigabytes is made
sunrunner20not quota
SweetAndLowto recover from 100% full you just truncate a large file
sunrunner20doesn't always work SweetAndLow
PLCI use my Freenas like SSH Forward
SweetAndLowsunrunner20: example of when it does't? cuz it should work
PLCso now, I'm looking for a failure in system … not sure it's really my local worksation ...
sunrunner20SweetAndLow, I think that fails if you have snaps
SweetAndLowsunrunner20: ahh yeah snaps will not get deleted
SweetAndLowand you won't be able to delete teh snap
SweetAndLowgotcha yeah that makes sense
sunrunner20and lets be honest if you aren't using snaps on ZFS you're doing it wrong
sunrunner20I forget if you can delete snaps with a completely full volume or not
sunrunner20I wan't to say you can't
sunrunner20but its been so long i'm not sure
SweetAndLowany delete operation will fail if your volume is full
SweetAndLowthey might have some room set aside to always do smaller deletes
PLCnothing to see ^^
PLCso no problem
PLCbye :)
Zewwyit actually succeeded
ZewwyOn 11.1 the only warning is - New feature flags are available for volume
ZewwyRefer to the "upragding a ZFS Pool" section
ZewwyI'm assuming if I was going to rebuild my pool (I am) any new pool will already be on the leastest n greatest
ZewwyHow does one view vDevs
ZewwyI guess Storage -> Select the Pool -> View Status
ZewwyVolume Status*
sunrunner20Zewwy, yes
sunrunner20and I haven't EVER updated the feature flags
sunrunner20I don't think they offer me anything
sunrunner20AllanJude, does resumable send/recv need a feature flag?
AllanJudefeature flags are for on-disk changes
AllanJudeit just needs both sides to support it
AllanJudeand you have to request it on the recv side when you start the send/recv
AllanJudewith the -s flag (save the partially received stream)
sunrunner20don't think freenas is setup for that yet
sunrunner20hrm, how does it know where to restart the stream and check for validity?
sunrunner20if my understanding is right the validity can be verified because its a partial ZFS filesystem so it should still have the b-tree checksums
sunrunner20AllanJude, don't say it enough- You're awsome
AllanJudeso, on the recv side
AllanJudeit makes a dataset property called resume_token
AllanJudewhich is a long text string
AllanJudewhich encoders the source dataset, snapshot you were sending, and the object id of the item it last received
AllanJudeso you just do 'zfs send -t <token> | zfs recv -s destination
AllanJudeto resume
AllanJudenote: you must remember to include other flags, like -L or -c
AllanJudeelse it will not use them going forward
AllanJudewhich can catch you off guard
AllanJude(-L is large blocks, for 1mb records, and -c is to send compressed blocks without decompressing them)
AllanJudenote: zpool upgrade only takes a few seconds, and is not a major operation really, it just enables the new features and does some top level metadata updates if required. The only reason not to do it is to still be able to import the pool on older systems that don't support those features
sunrunner20I was going to google that and not bother to make you tell me what a switch did lol
AllanJudesome features are read-only compatible, or only activate when you actually use it
AllanJudeACTION forgot the -L once, and had to resent 8TB of data
joe42Is it typical to have non-matching serial numbers when comparing output of sas2ircu & smartctl or other commands ? I am trying to find a streamlined way of identifying disks in our JBODs
AllanJudelike, I started with -L, and then resumed, and forgot to include it
AllanJudejoe42: if it is a proper jbod, try 'sesutil map'
sunrunner20I've not been upgrading my pool so I could go back
AllanJudeit uses SCSI Enclosure Services, to generically be able to blink the locate/fault LEDs in your chassis or JBOD
AllanJudeit works with pretty much any controller, instead of being vendor specific like sas2ircu
AllanJudeand it also tracks 'da17 is in slot #12 on JBOD #2'
joe42AllanJude: Thanks. That seems to match slot number and Dev name.
AllanJudeACTION helped write that, with
AllanJudebecause, there are 60 disks in my machine, 36 in the chassis, and 24 in the jbod
AllanJudeand being able to find out where they are is super helpful
AllanJudeand being able to blink the LED with just 'sesutil locate da17 on'
AllanJudeis very helpful
sunrunner20I have to make do with serials :|
AllanJudeor being able to turn the LED on for an empty slot, to tell the rack monkey where to install the new drive
AllanJudeI also use serials
AllanJudebut, having both is super helpfu.
sunrunner20at least on my main server
sunrunner20my actual server that comand might work
AllanJudeone of my pools:
AllanJudefront slot #00
AllanJudeback slot
sunrunner20actual server is going to have 16 drives in it this weekend
AllanJudesled 1 front slot #, seld1 back slot #
sunrunner20gpt labels?
joe42This is the pool I am currently working with. We have several of these servers running freenas now.
axsuulSo I flashed my IBM ServeRAID M1015 SAS/SATA controller to IT mode and to P20. In my UEFI shell I verified it with `sas2flash -listall`. However, once I boot into FreeNAS, running `sas2flash -listall` in terminal returns "No LSI SAS adapters found! Limited Command Set Available!". What gives?
axsuulI'm on FreeNAS 9.10
Cpuroastyou did something wrong
Cpuroastaxsuul: did you do the DOS part of it?
axsuulCpuroast : yep, I erased the firmware in DOS mode
axsuuldidn't receive any errors in DOS or UEFI
Cpuroastyou did the megarec procedure
Cpuroastthen the sas2flash procedure
axsuulYep, `megarec -writesbr 0 sbrempty.bin`, and `megarec -cleanflash 0`
axsuulin DOS
axsuulthen rebooted into UEFI
axsuuland ran `sas2flash.efi -o -f 2118it.bin`
axsuulfollowed by `sas2flash.efi -o -sasadd 500605B003D027C0`
Cpuroastand sas2flash.efi -list doesn't see it
axsuulCpuroast: it sees it in UEFI, shows correct version and everything
Cpuroastwhat does sysctl -a | grep mps
axsuulCpuroast :
Cpuroastit's online dev.mps.0.firmware_version:
Cpuroastdoes freenas see drives
axsuulhmm then why don't I see my disks
Cpuroastyou bought a breakout cable
Cpuroastor are you using a backplane
axsuulno backplane, breakout cable
Cpuroastdid you get a forward breakout cable
Cpuroastor a reverse breakout cable
axsuulGot 2x of these
Cpuroastyes they look the same, but no they aren't
SweetAndLowaxsuul: that error usually means things are still in IR mode
Cpuroastok, breakout cables are good
axsuulSweetAndLow: How can I check?
SweetAndLowthe thing is you are not in IR mode, you have flashed to IT mode
Cpuroastgo back to DOS or UEFI
SweetAndLowso it's a super strange error
Cpuroastand see what it says
Cpuroastwith sas2flash -list
Cpuroastif you had flashed mptsas2.rom
Cpuroastyou could have loaded the card's BIOS
Cpuroastand seen if it sees the drives
axsuul-1Ok, let me check
Cpuroastsas2flash.efi -b mptsas2.rom
Cpuroastmptsas2.rom is in the same zip package
Cpuroastas 2118it.bin
axsuul-1Alright, here's my output of `sas2flash.efi -list`
axsuul-1Thanks for your help btw :)
Cpuroastlooks good, just BIOS is missing
axsuul-1Cpuroast: So I should run `sas2flash.efi -o -f 2118it.bin -b mptsas2.rom`?
Cpuroastfirmware is already in
Cpuroastsas2flash.efi -b mptsas2.rom
Cpuroastshould do it
axsuul-1Cpuroast: look good?
Cpuroastyou'll see the LSI BIOS load at boot
axsuul-1Cool, what's interesting is the instructions i followed did list those BIOs steps but he/she said it wasn't necessary
Cpuroastthe BIOS is optional, but I like seeing it
Cpuroastgives you a pre-boot confirmation things are correctly detected
Cpuroasteven before you have an OS installed
axsuul-1ah gotcha
axsuul-1Cpuroast: still no drives, doh
Cpuroastdo they list them in the card's BIOS
Selavido they have power and spin up?
Cpuroastyou can access it with a key-combo
Cpuroastwhen you see the LSI BIOS appear
mrelceeACTION has flashbacks to 2016 building an efi boot flash forthat
Cpuroastmodern boards come with EFI Shell natively
Cpuroastso you don't actually need an external one
Cpuroastaxsuul-1: if the LSI BIOS doesn't see them and the drives are actually correctly connected and spinning
Cpuroastmaybe they sent you the wrong breakout cables
Cpuroastyou said you bought 2 of them are you using both?
Cpuroastas in you have 8 drives
Cpuroastyou could try swapping the breakout cables around
Cpuroastas in try 1 at a time
Cpuroaston 1 drive
Cpuroastsee if it shows up
axsuul-1Cpuroast: thanks will try that
Cpuroastif they sent you reverse breakout cables by accident
Cpuroastthey won't work
sunrunner20I hate how there's backwards and forwards breakout cables for SAS
Cpuroastsunrunner20: it's life :)
Cpuroastthey both have their uses
sunrunner20oh I understand
sunrunner20I just think with a little more work they could have eliminated it
Cpuroastnormally SAS cards are only used with straight cables
Cpuroastwith backplanes
Cpuroastbreakout cables were an afterthought
sunrunner20ffs we figured out how to put color ontop of a black and white signal without breaking compatability
Cpuroastreverse ones are rarely used
sunrunner20granted we got to cheat for that
Cpuroastthere;s way more forward units out there
mgolischwhat are they used for?
Cpuroastthen reverse ones
Cpuroastmgolisch: breakout cables?
sunrunner20iirc reverse ones are used to combine like 4 regular SAS ports into one sff8087 port
Cpuroastfor backplane usage
sunrunner20almost never happens
sunrunner20like Cpuroast was saying
mgolischi see
Cpuroastthere was a popular supermicro board
Cpuroastwhich had a 2308 onboard
Cpuroastwith discrete SAS ports
Cpuroastessentially had a forward breakout cable onboard
Cpuroastso you need a reverse one to undo the forward breakout cable
Cpuroastso you can connect it back to a backplane
sunrunner20I love this app /s
sunrunner20since mid 2014
sunrunner20it has created 100gb of jpegs
sunrunner20for UAT environment
sunrunner20the jpegs shouldn't live for more than 30 days
sunrunner20but this POS doesn't do anything right
sunrunner20I just found a section where it builds a string and assigns it to a variable then immediately reassigns that variable to another one. Never uses the first one again
FreenasguyHey so i am replicating a couple pools to another freenas box, on the source the total pool used size is 1.9TB. On the target it reports 5.2TB, is this normal?
FreenasguySame this with 2 other storage pools that are being replicated
oozboozcompression? de-duplication?
sunrunner20I was thinking snapshots
Freenasguylz4 compression, no dedup
FreenasguyPools are VM storage
sunrunner20well I'll be back later folks
sunrunner20time to go home
FreenasguySorry i should have been clear, a couple volumes, not pools
mgolischso its filesystems?
clustyis there a way to convert a zfs dataset from case sensitive to case insensitive ?
FreenasguyThe vols are storing VMware files
mgolischbut its filesystems not zvols?
FreenasguyThey are datasets in the same pool
Freenasguynot zvols
mgolischi see
FreenasguySet to replicate hourly and keep 2 weeks of snaps
mgolischmaybe compare snapshots on source and dest
Freenasguythe secondary freenas reports almost 25k snapshots total
FreenasguyWith each one taking probably 1-2gb used space according to the us
Freenasguy"Delete stale snapshots on remote system" is not set
Freenasguythat'd probably do it
m0nkey_^^ :)