| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-12-19 | |||
| 12:18:07 | ebbex | I've created a server with swap, where both root and swap are rbd, yet I have a "huge" swap-file on my compute node under nova/instances/_base/, and I see in the logs a "nova-rootwrap touch -c ...ova/instances/_base/swap_16384" going off about once a minute on that compute node. Where can I read up on the code that creates that file, and how long is the swap-file supposed to stay there? | |
| 12:19:17 | jaypipes | ebbex: the swap file should stay there for the life of the VM (since it's the swap content for the image...) | |
| 12:20:05 | jaypipes | ebbex: though I'm not sure why you'd see the touch -c command show up more than once. that's odd... | |
| 12:22:04 | ebbex | "virsh domblklist instance-00000041" gives: vdb vms/6e366e4b-2d87-48ac-a99c-999706e7e4f0_disk.swap, (on the ceph cluster) which I take it is where the instance gets to write swap to, right? No actual writes going to the _base/swap | |
| 12:24:20 | ebbex | I'm afraid that we might end up with a full disk thanks to swap images on our computenode as we have really small disks there. Yet vast amounts of storage on ceph. | |
| 12:25:57 | jaypipes | ebbex: hmm, I'm not sure. sure... mdbooth you around? | |
| 12:26:01 | lyarwood | mdbooth: ^ that smells like a bug, looksing at the code the fetch_func for creating swap is always _create_swap in nova/virt/libvirt/driver.py | |
| 12:26:10 | lyarwood | looksing | |
| 12:26:11 | lyarwood | :| | |
| 12:26:16 | jaypipes | mdbooth: does swap file get fulfilled by local disk even when ceph is used? | |
| 12:26:22 | jaypipes | oh, hey lyarwood :) | |
| 12:26:33 | lyarwood | \o_ morning | |
| 12:28:45 | ebbex | jaypipes: Yeah, I think it's kinda odd touching the file every minute, if the ImageCache tries something like _remove_old_enough*. I don't really understand how it's all supposed to hang together. | |
| 12:29:40 | jaypipes | ebbex: we've sent up the bat-signal for mdbooth :) hopefully he can share his insight on this (I'm afraid I'm not proficient enough in this area of the codebase) | |
| 12:30:41 | ebbex | :) | |
| 12:42:53 | jaypipes | stephenfin: I'd just go ahead and take over that InstancePCIRequest patch... ralonsoh, you cool with that? | |
| 12:47:01 | ralonsoh | jaypipes, stephenfin: but I've been taking care of my remaining patches. Anyway, if doing this we can have this 8 months patch merged, is ok | |
| 12:48:08 | openstackgerrit | Claudiu Belu proposed openstack/nova master: tests: autospecs all the mock.patch usages https://review.openstack.org/470775 | |
| 12:52:46 | jaypipes | ralonsoh: cool. it's just that stephenfin has been making some other changes around InstancePCIRequest object to support NUMA PCI affinity policy, so I thought it would be easier to have him take it over. | |
| 12:54:00 | jaypipes | gibi: are you planning on pushing a patch around that test_server_groups.py bug? | |
| 13:07:14 | gibi | jaypipes: yes, I'm working on that right now | |
| 13:07:26 | jaypipes | gibi: cool. | |
| 13:12:15 | openstackgerrit | Merged openstack/nova master: Deprecate configurable Hide Server Address Feature https://review.openstack.org/526297 | |
| 13:30:41 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Fix false positive server group functional tests https://review.openstack.org/529063 | |
| 13:30:45 | gibi | jaypipes: ^^ | |
| 13:30:55 | jaypipes | cool, thanks | |
| 13:34:39 | cdent | gibi: since you've spent a lot of time in the servers functional tests, can you recall how good the coverage is for the various migrations? I'm hoping that existing tests cover https://review.openstack.org/#/c/528089/ | |
| 14:02:01 | bauzas | mmm, blaming libvirt/driver.py seems a bit habit :p | |
| 14:02:10 | bauzas | s/bit/bad | |
| 14:10:31 | gibi | cdent: I think it is safe to assume that the changes in https://review.openstack.org/#/c/528089/ is covered with the existing functional tests | |
| 14:11:01 | gibi | cdent: I put the patch on my review list | |
| 14:11:11 | cdent | thanks gibi | |
| 14:11:59 | mriedem | edleafe: the failure on https://review.openstack.org/#/c/511358/ is because we aren't removing the existing allocations for the instance (from the tried and failed host) before we try allocating resources on the alternate | |
| 14:12:07 | mriedem | so the report client thinks we're doing a move operation, which we aren't | |
| 14:13:53 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Fix false positive server group functional tests https://review.openstack.org/529063 | |
| 14:15:02 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Fix false positive server group functional tests https://review.openstack.org/529063 | |
| 14:15:18 | mriedem | _move_operation_alloc_request is broken if we get the allocation candidates using 1.12 | |
| 14:17:10 | edleafe | mriedem: ok, just settling in. Will look over that shortly | |
| 14:20:16 | mriedem | i'll open a bug for the _move_operation_alloc_request thing | |
| 14:23:55 | mriedem | https://bugs.launchpad.net/nova/+bug/1739042 | |
| 14:23:55 | openstack | Launchpad bug 1739042 in OpenStack Compute (nova) "_move_operation_alloc_request fails with TypeError when using 1.12 version allocation request" [Undecided,New] | |
| 14:24:06 | kashyap | mriedem: When you get a moment, my 'logstash' foo isn't helping me; I want to see how many times this error has occurred: "error: connection closed due to keepalive timeout" | |
| 14:24:11 | kashyap | Putting it verbatim here http://logstash.openstack.org/#/dashboard/file/logstash.json | |
| 14:24:14 | kashyap | Didn't help. | |
| 14:24:23 | mriedem | kashyap: where is it originating from? | |
| 14:24:27 | kashyap | mriedem: stable/newton | |
| 14:24:32 | mriedem | which file? | |
| 14:24:33 | kashyap | Let me get a link | |
| 14:24:45 | kashyap | mriedem: There - http://logs.openstack.org/38/528338/4/check/legacy-tempest-dsvm-multinode-live-migration/d867726/job-output.txt.gz#_2017-12-18_20_20_56_230121 | |
| 14:24:55 | kashyap | It's this one: LiveMigrationTest.test_live_block_migration_paused | |
| 14:25:18 | mriedem | i don't see "error: connection closed due to keepalive timeout" in there at all | |
| 14:25:20 | kashyap | I debugged it a bit this morning w/ upstream libvirt & QEMU folks. And I'm setting up a reproducer to see if I can get to it | |
| 14:25:33 | kashyap | mriedem: Ah, sorry; that error actually comes from libvirtd log, let me get that link | |
| 14:25:40 | mriedem | we don't index the libvirtd logs | |
| 14:25:44 | mriedem | which is why it's not in logstash | |
| 14:25:49 | lyarwood | it's also in n-cpu FWIW | |
| 14:26:09 | lyarwood | http://logs.openstack.org/38/528338/4/check/legacy-tempest-dsvm-multinode-live-migration/d867726/logs/subnode-2/screen-n-cpu.txt.gz#_2017-12-18_20_20_13_894 | |
| 14:26:10 | kashyap | mriedem: There - http://logs.openstack.org/38/528338/4/check/legacy-tempest-dsvm-multinode-live-migration/d867726/logs/subnode-2/libvirt/libvirtd.txt.gz#_2017-12-18_20_20_13_880 | |
| 14:26:14 | kashyap | Ah-ha | |
| 14:26:21 | kashyap | mriedem: Any reason we don't index it? | |
| 14:26:24 | mriedem | http://logs.openstack.org/38/528338/4/check/legacy-tempest-dsvm-multinode-live-migration/d867726/logs/subnode-2/screen-n-cpu.txt.gz#_2017-12-18_20_20_13_894 is debug | |
| 14:26:29 | mriedem | we index INFO+ | |
| 14:26:36 | mriedem | we don't index libvirtd because it kills the indexer | |
| 14:26:39 | mriedem | too much content | |
| 14:26:55 | kashyap | Okay, the screen-n-cpu.txt has it | |
| 14:26:57 | lyarwood | mriedem: it's also above in ERROR | |
| 14:27:06 | kashyap | Yeah, it's in ERROR | |
| 14:27:11 | mriedem | http://logs.openstack.org/38/528338/4/check/legacy-tempest-dsvm-multinode-live-migration/d867726/logs/subnode-2/screen-n-cpu.txt.gz#_2017-12-18_20_20_13_893 | |
| 14:27:15 | mriedem | ok that should work | |
| 14:27:40 | kashyap | mriedem: Do you kow how could it kill the index? Due to its size? | |
| 14:27:45 | mriedem | kashyap: yes | |
| 14:27:47 | mriedem | size | |
| 14:28:06 | mriedem | http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22Live%20Migration%20failure%3A%20internal%20error%3A%20connection%20closed%20due%20to%20keepalive%20timeout%5C%22%20AND%20tags%3A%5C%22screen-n-cpu.txt%5C%22&from=7d | |
| 14:28:16 | kashyap | Ah, interesting, even ~217K is too much? | |
| 14:28:21 | kashyap | (Anyway, that's fine.) | |
| 14:28:50 | kashyap | mriedem++ | |
| 14:28:53 | mriedem | kashyap: it's that times however many jobs we run PER DAY | |
| 14:28:54 | kashyap | No Karma bot | |
| 14:29:56 | kashyap | Okay, only 2 hits so far; thanks mriedem | |
| 14:30:05 | mriedem | yeah, but all on newton | |
| 14:30:12 | kashyap | Right | |
| 14:30:16 | mriedem | so i'm guessing it's related to the version of libvirt/qemu we have in newton | |
| 14:30:54 | kashyap | (Don't know the root cause of it yet; could be QEMU, could be libvirt. There are 2 other bugs filed for them - https://bugzilla.redhat.com/show_bug.cgi?id=1367620) | |
| 14:30:54 | openstack | bugzilla.redhat.com bug 1367620 in libvirt "storage migration fails due to keepalive timeout" [High,Assigned] - Assigned to jdenemar | |
| 14:30:54 | mriedem | i don't think we're using UCA packages in newton jobs | |
| 14:31:24 | mriedem | ^ is libvirt 1.3.1 and qemu 2.5 | |
| 14:31:29 | mriedem | we're way newer than that on queens | |
| 14:31:37 | kashyap | Yeah, saw the versions earlier in the day | |
| 14:31:55 | kashyap | Is it worth it to use UCA in that case? Maybe not, for these rare one-off cases | |
| 14:32:13 | mriedem | not at this point for newton | |
| 14:32:31 | kashyap | Yep, noted. | |
| 14:32:47 | mriedem | bauzas: this is a regression introduced in newton https://review.openstack.org/#/c/528835/ - would be good to get your review on that | |
| 14:33:12 | openstackgerrit | Bernhard M. Wiedemann proposed openstack/nova master: Fix 4 doc typos https://review.openstack.org/529084 | |
| 14:35:41 | bauzas | mriedem: ack, looking | |
| 14:36:29 | bauzas | mriedem: ah, good call | |
| 14:37:22 | bauzas | I remember we had a shit number of races for the BuildRequest object | |
| 14:38:48 | edleafe | mriedem: about the func test failure: this line should de-allocate against the instance: https://review.openstack.org/#/c/511358/43/nova/compute/manager.py@1778 | |
| 14:38:57 | mriedem | right so what was added there in newton was just for novalidhost on the initial create | |
| 14:39:01 | mriedem | but didn't take into account reschedules | |
| 14:39:09 | mdbooth | mriedem: Any chance you could have another look at the BDM uuid patches? https://review.openstack.org/#/c/242602/25 and the following 2 are the ones which do the db modification. I addressed your review comments. | |