SargunWhere do BTRFS_EMPTY_SUBVOL_DIRs come from?
MoKnorrie: What did you mean with deleting snapshots with lot of snapshots may become painful? Isn't the delete actually done in the background, so that command would not block for a while?
darklingIt doesn't block, but it does soak up I/O bandwidth.
Modarkling: Where will I see that? iotop? Will that be btrfs-cleaner process?
darklingI don't know, I'm afraid.
Jay2k1you will see it in iotop
Jay2k1Mo: you'll have btrfs-cleaner with lots of IO in iotop for like half of the time, and the other half it'll be btrfs-transacti with 100% cpu in top
btrfs076hi, how can I guaranty autodefrag on external HDD when auto mounted in any machine I plug it, well, is it possible?
KeI don't think it is, but such volume properties have been discussed
KnorrieZygo: ah the bulk patch is about csum deletion, which of course is related to removing data that is not referenced anymore when throwing out snapshots
Knorrieinteresting
btrfs076Ok, thanks, another thing I'm curious is if there is any work going on native encryption.
ivo_hey
ivo_one quick question ... I am using the dup profile on singles ssd
ivo_still getting uncorrectable errors ... but more of them
ivo_after reboot everything is fine
ivo_no smart issues
ivo_any Ideas?
frinnstkernel version?
optymemory?
frinnsthave you run a scrub?
ivo_ 4.4.57-18.3-default #1 SMP Thu Mar 30 06:39:47 UTC 2017 (39c8557) x86_64 x86_64 x86_64 GNU/Linux
ivo_32GB ECC Memory
ivo_yes, I get the errors when I run the scrub
Keyour ssd can use its radical freedom to function incorrectly until reboot
Keit would not be impossible to trace what you are reading, if the errors are frequent
ivo_actually I don't know if the reboot fixes or the clear_cache mount option
ivo_will verify this the next time
Keclear cache definitely should do nothing of the sort
ivo_I've recently starting using scrub :)
ivo_ok, so you don't think is a hardware issue??
ivo_after the reboot everything is fine
Ke(16:05:09) < Ke> your ssd can use its radical freedom to function incorrectly until reboot
ivo_the other Filesystem seems to be always fine it uses 2 ssd in raid0
Kethat would definitely be me saying it could be a hw problem
Keivo_: anyway, if you see scrub errors something is definitely wrong and you should not ignore them
optywhat ssd model?
ivo_samsung 850 Pro
ivo_256Gb
ivo_it is the system drive
optypro should be okay (i heard)
ivo_what do you mean?
ivo_I use the 512GB Pros in RAID0 for my workspace
ivo_so now I've run it again twice and there are errors again :D
ivo_corrected errors: 0, uncorrectable errors: 182, unverified errors: 0
ivo_and the number is getting bigger
ivo_:D
multicoreivo_: dev stats?
ivo_https://paste.gnome.org/pky4k04zo
multicorebetter start checking your hw
ivo_but why everything is fine after reboot?
ivo_no errors found ..
multicoreafter reboot scrub reports no errors?
ivo_ohhh noooo
ivo_It is a Windows 7 VM
ivo_when this VM is running there are errors while scrubbing
ivo_when the machine is closed no errors ...
ivo_so strange ... because the VM is on the other FS :S
darklingAt a guess, it's probably using O_DIRECT, where the checksums don't always immediately match what's written as data.
ivo_ok .... this will explain it
darklingSo if you're scrubbing at the same time as something's writing to the VM image, the VM image will look like it's got csum errors, but they'll catch up eventually.
multicore"because the VM is on the other FS"
ivo_thanx, now I can learn and sleep in peace
multicorenot a DIO issue then
ivo_the virtual drives are stored on the other FS, but probabnly some temporary stuff is saved on the /
multicoreok
Kealso nocow files do not have checksums
ivo_I'll try it a couple of times just to be sure
ivo_it is the vm
ivo_while it runs ... errors, when it shuts down no errors
SargunThe mapping of subvols:qgroups is always 1:(0,1), correct?
SargunAny ideas what might cause this:
Sargunroot@ubuntu:~# dd if=/dev/zero of=/mnt/btrfs/subvol/ack bs=1
Sargundd: failed to open '/mnt/btrfs/subvol/ack': Disk quota exceeded
Sargunwhen: root@ubuntu:~# sudo btrfs qgroup show /mnt/btrfs/subvol -reF shows:
Sargunqgroupid rfer excl max_rfer max_excl
Sargun0/259 16.00KiB 16.00KiB 1.46GiB 150.00MiB
SargunIt's not getting passed start_transaction:
Sargun 1) | btrfs_start_transaction [btrfs]() {
Sargun 1) 2.243 us | btrfs_qgroup_reserve_meta [btrfs]();
Sargun> Bugs in accounting code might cause false out of space situations.
kdaveKnorrie: the nossd gif almost got me hypnotized, what do the blue spots mean?
Knorrie"The black parts are unallocated disk space. Raw disk space that is allocated to be used for data (white), metadata (blue) or system (red) gets brighter if the fill factor of block groups is higher."
kdaveso it's filling all the free space, in the left lower corner, and it takes long, that's what you're pointing out, right?
Knorrieyes, nossd helps to fill them up more, and the fragmentation based feeding of blockgroups I run now also helps a lot
Knorrielatest work in progress experiment https://github.com/knorrie/python-btrfs/blob/fragmented/examples/fragmentation3.py
Knorrieit gives small and big free space fragments a low score, and medium sized ones a high score
Knorriewhich turns out to work really well
Knorriehttps://syrinx.knorrie.org/~knorrie/btrfs/shift.jpg
kdavemath!
Knorriebecause if you have 1000 4kiB free space fragments, it's only 4MiB in total, that's not very interesting
Knorrieso for example this gets a high score https://syrinx.knorrie.org/~knorrie/btrfs/1491489581-000449-3153992-77-23924607811584.png
Knorrieand in practice it seems that when skipping block groups that are like >80 full or not too fragmented, and feeding the ones that are to balance, the data gets pushed down into the open spots that were just skipped
Knorrieso much more effective way of using balance than just doing dusage etc on a large fs
kdavehmm, sounds like a smarter filter
Knorrieyeah, at least it's nice that the kernel api allows experimenting to work towards something good without having to change the kernel
Knorriealso notice the convenience of calling balance ;] btrfs.ioctl.balance_v2(fs.fd, vstart=min_vaddr, vend=min_vaddr+1))
Knorrieanother thing I was thinking about is to get the list with "blacklisted" blockgroups from the kernel, do debug the ssd mode, which causes skipping a lot of them for writes
Knorriemaybe first with debugfs or something
Knorrieand then look at the pictures generated from them and fragmentation score to see if the current "fragmented" flag is effective enough or not
Knorriea future idea could be to just feed that list internally to balance if you do balance with --do-whatever-seems-best-for-me
Knorrieignoring free space and going enospc on a half filled fs is a real end user problem now
kdaveyeah, the vstart/vend is enough to experiment with the heuristics
kdaveI'm a bit worried about the interface extensions if you need to supply additional information from user, eg. the blacklist
Knorrieno, that list is in the kernel
Knorriethe fragmented bool
kdaveaha, reading it again, I've reversed it
KnorrieI'm changing a bit of webserver config, so I can show all pics and output of the current process
SargunCan btrfs quotas be assigned to arbitrary directories?
kdaveSargun: no
kdavejust subvolumes
SargunSo, if I have 100 different users, or containers, I have to have 1 subvol per?
SargunI can't create a quota for /home, and then a child qgroup for each user
Knorriekdave: https://btrfs-fragmented.mendix.com/ see processed/ dir for everything that has been cleaned up in the last days
Knorrienow.html shows what's going on right now
Knorrieif this plowing over the whole fs finishes in a few weeks I can just do a little bit everyday to keep allocated/unused at a minimum
millertiIf I delete a subvolume, how long before the free space indication is updated?
Knorrieafter the cleaner in the background is done with it
Knorrieif you want to wait for that, e.g. in a script, do 'btrfs sub sync /mountpoint', which exits when it's done
millertiKnorrie: Thanks!
millertiKnorrie: Can I monitor progress instead of waiting?
Knorriethat's what I also wanted to do a week ago, and then I wrote something for that: :D
Knorriehttps://github.com/knorrie/python-btrfs/commit/dd34044adf24f7febf6f6992f11966c9094c058b
millertiKnorrie: Cool, tanks.
millertican't type. :)
potty-nyanwhat's the best way to go about using subvolumes in a mount point that already contains a directory structure?
Knorrieif you want to convert instead of mixing it... you can snapshot the toplevel to a new subvolume and then just rm -rf everything else (make a readonly snapshot first if you want to make sure you don't accidentally remove everything) :]
millertiWell, the subvolume deletes are making progress...
potty-nyanKnorrie: ok. i will btrfs send the snapshot to another computer before i rm -rf
Sargunkdave: Would it make sense to add some of the macros, or helper functions to iterate over objects (search_ioctl) in libbtrfs
kdaveSargun: yeah, that would be good for the library and also could be used internally, as the code is more or less copy&paste
Sargunkdave: I've been trying to make the search ioctl better
Sargunkdave: One of the things I've thrown around, and mentioned to cmason is having a bpf program to fold over the tree
kdaveSargun: so you're talking about the kernel code? because libbtrfs is userspace in my view
schoppenhauerhi. is it possible to have a readonly snapshot and another readonly snapshot from the same volume, and then get a binary diff between both? I heard something around btrfs-send and btrfs-receive.
schoppenhauerI actually want to create a setup where I can make backups through the network
schoppenhauerI currently do incremental backups using rsync, but I have some virtualbox-appliances running …
Knorriehttps://btrfs.wiki.kernel.org/index.php/Incremental_Backup
schoppenhauerKnorrie: thx
Sargunkdave: there was a context switch there
Sargunthat I didn't state
KnorrieI think it's a bit unclear if the middle statement applies to the first, the third or both :]
mint_hello?
potty-nyanhi
Knorriemoo
mint_Has anybody installed linuxMint18.1xfce on a btrfs system?
mint_my installation keeps failing, even with different options
Knorriethen you need to find the place where the errors about what's wrong get logged
mint_moo... I see
baudotACTION pricks up her ears
mint_I guess nobody has?
specinghave you tried asking in mint's channel?
mint_yeah
mint_only to be trolled by cryptodan-portable
specinginstalling gnu/linux is most often ridiculously easy
mint_and arrogantly told to bugger off and find another channel to ask these kinds of questions
specingso you can install it to extX
specingrsync to btrfs
specingand change fstab
mint_Yes, I've installed gnu/linux a number of times since the late 90s
Knorriewell, your question is if someone ever did something, and the answer to that question will probably not help you further
mint_and the latest I've used is Ext4
mint_oh wait, we can convert rsync to btrfs???
Knorriedoes someone in the world drive a Citroen? Mine doesnt work
mint_The initial question I asked there was "Are there any instructions to installing Linux Mint onto a BTRFS partition properly?"
Knorriethe answer to that would be "did you lookup the mint manual"
mint_and because I said "thanks ^_^" to a response, cryptodan-portable thought I was a troll (since "emoticons are immature") rofl
mint_yes,
mint_i looked up the "mint manual"
Knorrieif it's not in there, then mint people maybe don't care about it
mint_thing is, I can't find much on Mint18 and BTRFS
mint_It just says that it supports BTRFS officially and stops at that lol
potty-nyanare you installing to an ssd or hdd?
Knorriemint is a convenience distro which is I think an ubuntu clone
mint_SSD
potty-nyanwhat mount options are you using for the btrfs partition?
mint_Yes, Mint is a polished distro downstream from Ubuntu
mint_mount option "/"
Knorrieso maybe you need to do a step back and look at ubuntu or even debian instead to get more support for doing non-standard things
mint_and a primary partition for /boot
mint_Yeah, the literature for ubuntu and debian are several years old. Maybe I need to keep digging (2 weeks in of diggin)
mint_I'm probably looking in all the wrong places... -_- Maybe I'll just go back to Arch ...
mint_and stick with Ext4
potty-nyanthere are some ssd-specific mount options for btrfs that may or may not be causing hiccups
potty-nyanrelatime,ssd,space_cache
Knorrieyou still haven't mentioned what the actual thing is that fails, except for "it fails", which doesn't tell much
darklingYou may get some different answers if you can describe the nature of the failures.
mint_You might have a point there... I'd need to look more into that
potty-nyandoes the installer just hang?
mint_no, it brings up a prompt, though not sure where it dumps the logfile
Knorriewhat is 'it'?
specing> < mint_> oh wait, we can convert rsync to btrfs???
mint_it occurs in the middle of installation, I'll do it again
specingLOL
mint_it = the installer
Knorrieok
mint_specing, is it possible?
Knorriejust after which step, what happens just before that?
mint_I thought we had to format the partition to BTRFS prior to installation
potty-nyanwell it should work but
potty-nyanthis sounds like an issue with mint
potty-nyanmaybe the live usb kernel doesn't have btrfs drivers
mint_Yeah, it does... maybe I'll take Fed25 for a spin... check out Wayland and see if BTRFS flies well with it
potty-nyantry installing on ext4 then converting to btrfs afterwards
mint_Hm... so if the live USB version may not have it, I should try a different method
mint_I'll google how to convert an ext4
mint_I usually like starting scratch, but if converting is just as viable, then I'll consider that alley
potty-nyanjust make sure that once your kernel and programs are up to date it actually has btrfs drivers
mint_thanks potty-nyan, I'll look into that. I wonder where all the chatter is for Mint... or if I should sample another distro instead
mint_just wondering, what are you guys running?
KnorrieACTION debian
potty-nyani use funtoo
mint_cool
mint_debian looks tempting
mint_though, when I tried it back in early 2000s it wasn't a fun ride
potty-nyani would just stick to ubuntu if you want a binary distro
mint_ can't say that "I 'want' a *binary* distro" in particular...
Knorriemint_: I think you shouldn't base your decisions on an experience more than a decade ago
mint_any suggestions? :) I mean I've read about all the major and submajor distros out there, tried a handful,
mint_hahaha yes Knorrie... you're right about that. I'm sure deb has improved much since then.
mint_I'll check it out :D
KnorrieI run it since woody with boot floppies on a 486 with 2 or 4 MB of memory
mint_Oh the good ol' floppy days
mint_damn... "Installer Crashed: We're sorry; the installer crashed. After you close this window, we'll allow you to file a bug report using the integrated bug reporting tool. This will gather information about your system and your installation process. The details will be sent to our bug tracker and a developer will attend to the problem as soon as possible."
Knorrieright
specingmint_ Devuan
specingdebian without systemdumb
mint_crashed whyle installing the mint /plugininstall.py: ubiquity.install_misc.Installsteperror: GrubInstaller failed with code 1
mint_lol
mint_systemdumb
mint_Devuan? :/
potty-nyanis your /boot partition also btrfs?
mint_no, ext4
mint_and 500mb
mint_500MB*
potty-nyangrub supports ext4?
mint_grub2, I think
specingpotty-nyan: grub2 even supports btrfs
mint_oh no, yes, grub supports ext4 in mint
potty-nyanmy mind is blown
specingpotty-nyan: (and I'm using that support)
mint_Yeah, I feel ya potty-nyan , which is why I'm jumping ship from Arch and checking out Mint
mint_kinda large leap, but that's just my personality :/
potty-nyani always format /boot with ext2 out of habit
specingmy /boot is btrfs
mint_your /boot is btrfs? did you chattr it? or just install it as such
mint_specing:
specingits my normal rootfs, no special partition
mint_oh, so your boot is integrated with the / btrfs partition?
specingy
specingoh and GRUB2 also supports LVM and LUKS full disk encryption
mint_yup
mint_bbl... hopping off this live Mint disk and onto my Ext4 install
Knorriew00t again over 200GiB written without allocating a new chunk
Knorrienossd++
mpiechotkaHi. I'm trying to copy data on new system but when I try to use btrfs send I get "send ioctl failed with -25: Inappropriate ioctl for device"
Knorrietwo first things that come to mind is that either you're receiving on a fs that is not btrfs, or you might be using a mixture of 32 bit and 64 bit kernel and programs